[jira] [Commented] (CASSANDRA-13362) Cassandra 2.1.15 main thread stuck in logback stack trace upon joining existing cluster

[Thomas Steinmaurer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172779#comment-16172779
 ] 

Thomas Steinmaurer commented on CASSANDRA-13362:


Sorry for not replying earlier. At the end, it was an error on our side, 
introducing some kind of locking issue on STDOUT for the Cassandra process. I 
think this issue can be closed.

> Cassandra 2.1.15 main thread stuck in logback stack trace upon joining 
> existing cluster
> ---
>
> Key: CASSANDRA-13362
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13362
> Project: Cassandra
>  Issue Type: Bug
> Environment: 
>Reporter: Thomas Steinmaurer
> Attachments: td___2017-03-21-21-30-09.tdump, 
> td___2017-03-21-23-09-59.tdump
>
>
> Switching from Cassandra 2.0.17 to Cassandra 2.1.15 (DSC edition: 
> dsc-cassandra-2.1.15-bin.tar.gz) in a local VM based Linux environment for 
> installer verification tests.
> {noformat}
> [root@localhost jdk1.8.0_102]# lsb_release -d
> Description:  CentOS release 6.7 (Final)
> You have new mail in /var/spool/mail/root
> [root@localhost jdk1.8.0_102]# uname -a
> Linux localhost 2.6.32-573.el6.x86_64 #1 SMP Thu Jul 23 15:44:03 UTC 2015 
> x86_64 x86_64 x86_64 GNU/Linux
> {noformat}
> The test environment is started from scratch, thus in the following scenario 
> not an upgrade from 2.0 to 2.1, but a fresh 2.1 installation.
> The first node started up fine, but when extending the cluster with a second 
> node, the second node hangs in the following Cassandra log output while 
> starting up, joining the existing node:
> {noformat}
> INFO  [InternalResponseStage:1] 2017-03-21 21:10:43,864 DefsTables.java:373 - 
> Loading 
> org.apache.cassandra.config.CFMetaData@1c3daf27[cfId=a8cb1eb0-0e61-11e7-9a56-b20ca863,ksName=ruxitdb,cfName=EventQueue,cf$
> INFO  [main] 2017-03-21 21:11:11,404 StorageService.java:1138 - JOINING: 
> schema complete, ready to bootstrap
> ...
> INFO  [main] 2017-03-22 03:13:36,148 StorageService.java:1138 - JOINING: 
> waiting for pending range calculation
> INFO  [main] 2017-03-22 03:13:36,149 StorageService.java:1138 - JOINING: 
> calculation complete, ready to bootstrap
> INFO  [main] 2017-03-22 03:13:36,156 StorageService.java:1138 - JOINING: 
> getting bootstrap token
> ...
> {noformat}
> So, basically it was stuck on 2017-03-21 21:11:11,404 and the main thread 
> somehow continued on  2017-03-22 03:13:36,148, ~ 6 hours later.
> I have two thread dumps. The first from 21:30:
> [^td___2017-03-21-21-30-09.tdump]
> and a second one ~ 100min later:
> [^td___2017-03-21-23-09-59.tdump]
> Both thread dumps have in common, that the main thread is stuck in some 
> logback code:
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7fe93821a800 nid=0x4d4e waiting on 
> condition [0x7fe93c813000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0xc861bb88> (a 
> java.util.concurrent.locks.ReentrantLock$FairSync)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>   at 
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:224)
>   at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>   at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:217)
>   at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103)
>   at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
>   at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
>   at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273)
>   at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260)
>   at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442)
>   at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
>   at ch.qos.logback.classic.Logger.info(Logger.java:600)
>   at 
> org.apache.cassandra.service.StorageService.setMode(StorageService.java:1138)
>   at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:870)
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
>   - locked <0xc85d37d8> (a 
> 

[jira] [Updated] (CASSANDRA-13883) StrictLiveness for view row is not handled in AbstractRow

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13883:
---
Fix Version/s: (was: 3.11.1)
   (was: 3.0.15)
   (was: 4.0)
   4.x
   3.11.x
   3.0.x

> StrictLiveness for view row is not handled in AbstractRow
> -
>
> Key: CASSANDRA-13883
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13883
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> In {{AbstractRow.hasLiveData(nowInSecond)}}, it doesn't handle 
> {{strictLiveness}} introduced in CASSANDRA-11500. The {{DataLimits}} counts 
> the expired view row as live data and then the expired view row is purged in 
> {{Row.purge()}}. When query with limit, we will get less data.
> {code:title=test to reproduce}
> @Test
> public void testRegularColumnTimestampUpdates() throws Throwable
> {
> createTable("CREATE TABLE %s (" +
> "k int PRIMARY KEY, " +
> "c int, " +
> "val int)");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("mv_rctstest", "CREATE MATERIALIZED VIEW %s AS SELECT * 
> FROM %%s WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)");
> updateView("UPDATE %s SET c = ?, val = ? WHERE k = ?", 0, 0, 0);
> updateView("UPDATE %s SET val = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s SET c = ? WHERE k = ?", 1, 0);
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(1, 0, 
> 1));
> updateView("TRUNCATE %s");
> updateView("UPDATE %s USING TIMESTAMP 1 SET c = ?, val = ? WHERE k = 
> ?", 0, 0, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET c = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s USING TIMESTAMP 2 SET val = ? WHERE k = ?", 1, 
> 0);
> updateView("UPDATE %s USING TIMESTAMP 4 SET c = ? WHERE k = ?", 2, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET val = ? WHERE k = ?", 2, 
> 0);
> // FIXME no rows return
> assertRows(execute("SELECT c, k, val FROM mv_rctstest limit 1"), 
> row(2, 0, 2));
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(2, 0, 
> 2));
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13787) RangeTombstoneMarker and PartitionDeletion is not properly included in MV

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155526#comment-16155526
 ] 

ZhaoYang edited comment on CASSANDRA-13787 at 9/20/17 2:51 AM:
---

| source |  test  |   dtest |
| [trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-trunk] 
| [passed|https://circleci.com/gh/jasonstack/cassandra/571] | 
repair_tests.repair_test.TestRepair.simple_parallel_repair_test
repair_tests.repair_test.TestRepair.test_dead_sync_initiator
rebuild_test.TestRebuild.rebuild_ranges_test
repair_tests.repair_test.TestRepair.dc_repair_test
topology_test.TestTopology.simple_decommission_test
disk_balance_test.TestDiskBalance.disk_balance_decommission_test
repair_tests.repair_test.TestRepair.simple_sequential_repair_test
repair_tests.repair_test.TestRepair.thread_count_repair_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space|
| [3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.11] | 
[passed|https://circleci.com/gh/jasonstack/cassandra/573] 
|upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test |
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.0] | 
[passed|https://circleci.com/gh/jasonstack/cassandra/572] | 
rebuild_test.TestRebuild.simple_rebuild_test
global_row_key_cache_test.TestGlobalRowKeyCache.functional_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test
auth_test.TestAuth.system_auth_ks_is_alterable_test|

Failing dtests are either failing on trunk or passed on local.



was (Author: jasonstack):
| source |  test  |   dtest |
| [trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-trunk] 
| [passed|https://circleci.com/gh/jasonstack/cassandra/571] | {{passed on 
local:}}
repair_tests.repair_test.TestRepair.local_dc_repair_test
{{failed on trunk:}}
repair_tests.repair_test.TestRepair.simple_parallel_repair_test
repair_tests.repair_test.TestRepair.test_dead_sync_initiator
rebuild_test.TestRebuild.rebuild_ranges_test
repair_tests.repair_test.TestRepair.dc_repair_test
topology_test.TestTopology.simple_decommission_test
disk_balance_test.TestDiskBalance.disk_balance_decommission_test
repair_tests.repair_test.TestRepair.simple_sequential_repair_test
repair_tests.repair_test.TestRepair.thread_count_repair_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space|
| [3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.11] | 
[passed|https://circleci.com/gh/jasonstack/cassandra/573] | {{passed on local:}}
batch_test.TestBatch.logged_batch_doesnt_throw_uae_test
repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test
{{failed on 3.11}}
batch_test.TestBatch.batchlog_replay_compatibility_1_test
batch_test.TestBatch.batchlog_replay_compatibility_4_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test |
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.0] | 
[passed|https://circleci.com/gh/jasonstack/cassandra/572] | {{passed on local}}
batch_test.TestBatch.logged_batch_throws_uae_test
{{also failed on cassandra-3.0}}
offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
global_row_key_cache_test.TestGlobalRowKeyCache.functional_test
batch_test.TestBatch.batchlog_replay_compatibility_1_test
batch_test.TestBatch.batchlog_replay_compatibility_4_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test
auth_test.TestAuth.system_auth_ks_is_alterable_test|

Failing dtests are either failing on trunk or passed on local.


> RangeTombstoneMarker and PartitionDeletion is not properly included in MV
> -
>
> Key: CASSANDRA-13787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13787
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> Found two problems related to MV tombstone. 
> 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent 
> base rows are not shadowed in TableViews.
> If the range tombstone was not flushed, it was used as deleted row to 
> shadow new updates. It works correctly.
> After range tombstone was flushed, 

[jira] [Comment Edited] (CASSANDRA-13883) StrictLiveness for view row is not handled in AbstractRow

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172606#comment-16172606
 ] 

ZhaoYang edited comment on CASSANDRA-13883 at 9/20/17 2:03 AM:
---

| source | unit | dtest |
| 
[trunk|https://github.com/apache/cassandra/compare/trunk...jasonstack:CASSANDRA-13883-trunk?expand=1]|
 [passed|https://circleci.com/gh/jasonstack/cassandra/627] |   
repair_tests.repair_test.TestRepair.dc_parallel_repair_test
repair_tests.repair_test.TestRepair.dc_repair_test
repair_tests.repair_test.TestRepair.local_dc_repair_test
repair_tests.repair_test.TestRepair.simple_parallel_repair_test
repair_tests.repair_test.TestRepair.thread_count_repair_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test
auth_test.TestAuth.system_auth_ks_is_alterable_test
disk_balance_test.TestDiskBalance.disk_balance_decommission_test
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space
cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space|
| 
[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...jasonstack:CASSANDRA-13883-3.11?expand=1]
 | [passed|https://circleci.com/gh/jasonstack/cassandra/625] | 
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test
auth_test.TestAuth.system_auth_ks_is_alterable_test |
| 
[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...jasonstack:CASSANDRA-13883-3.0?expand=1]
 | [running|https://circleci.com/gh/jasonstack/cassandra/628]| 
global_row_key_cache_test.TestGlobalRowKeyCache.functional_test
repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test |
| 
[dtest|https://github.com/apache/cassandra-dtest/compare/master...jasonstack:CASSANDRA-13883?expand=1]
 |

CI looks good, I will restart a few flaky ones.

{code}
Changes:
1. Change {{AbstractRow.hasLiveData}} to check {{enforceStrictLiveness}}:  
if livenessInfo is not live and enforceStrictLiveness, then there is not live 
data.
2. For SPRC.group, use the first command to get {{enforceStrictLiveness}} 
since each command should be the same except for key.
{code}


was (Author: jasonstack):
| source | unit | dtest |
| 
[trunk|https://github.com/apache/cassandra/compare/trunk...jasonstack:CASSANDRA-13883-trunk?expand=1]|
 [passed|https://circleci.com/gh/jasonstack/cassandra/627] |   |
| 
[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...jasonstack:CASSANDRA-13883-3.11?expand=1]
 | [passed|https://circleci.com/gh/jasonstack/cassandra/625] |  |
| 
[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...jasonstack:CASSANDRA-13883-3.0?expand=1]
 | |  |
| 
[dtest|https://github.com/apache/cassandra-dtest/compare/master...jasonstack:CASSANDRA-13883?expand=1]
 |

CI looks good, I will restart a few flaky ones.

{code}
Changes:
1. Change {{AbstractRow.hasLiveData}} to check {{enforceStrictLiveness}}:  
if livenessInfo is not live and enforceStrictLiveness, then there is not live 
data.
2. For SPRC.group, use the first command to get {{enforceStrictLiveness}} 
since each command should be the same except for key.
{code}

> StrictLiveness for view row is not handled in AbstractRow
> -
>
> Key: CASSANDRA-13883
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13883
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> In {{AbstractRow.hasLiveData(nowInSecond)}}, it doesn't handle 
> {{strictLiveness}} introduced in CASSANDRA-11500. The {{DataLimits}} counts 
> the expired view row as live data and then the expired view row is purged in 
> {{Row.purge()}}. When query with limit, we will get less data.
> {code:title=test to reproduce}
> @Test
> public void testRegularColumnTimestampUpdates() throws Throwable
> {
> createTable("CREATE TABLE %s (" +
> "k int PRIMARY KEY, " +
> "c int, " +
> "val int)");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("mv_rctstest", "CREATE MATERIALIZED VIEW %s AS SELECT * 
> FROM %%s WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)");
> updateView("UPDATE %s SET c = ?, val = ? WHERE k = ?", 0, 0, 0);
> updateView("UPDATE %s SET val = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s SET c = ? WHERE k = ?", 1, 0);
> assertRows(execute("SELECT c, k, val FROM 

[jira] [Updated] (CASSANDRA-13883) StrictLiveness for view row is not handled in AbstractRow

[ZhaoYang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-13883:
-
Fix Version/s: 4.0
   3.11.1
   3.0.15
   Status: Patch Available  (was: In Progress)

> StrictLiveness for view row is not handled in AbstractRow
> -
>
> Key: CASSANDRA-13883
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13883
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> In {{AbstractRow.hasLiveData(nowInSecond)}}, it doesn't handle 
> {{strictLiveness}} introduced in CASSANDRA-11500. The {{DataLimits}} counts 
> the expired view row as live data and then the expired view row is purged in 
> {{Row.purge()}}. When query with limit, we will get less data.
> {code:title=test to reproduce}
> @Test
> public void testRegularColumnTimestampUpdates() throws Throwable
> {
> createTable("CREATE TABLE %s (" +
> "k int PRIMARY KEY, " +
> "c int, " +
> "val int)");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("mv_rctstest", "CREATE MATERIALIZED VIEW %s AS SELECT * 
> FROM %%s WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)");
> updateView("UPDATE %s SET c = ?, val = ? WHERE k = ?", 0, 0, 0);
> updateView("UPDATE %s SET val = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s SET c = ? WHERE k = ?", 1, 0);
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(1, 0, 
> 1));
> updateView("TRUNCATE %s");
> updateView("UPDATE %s USING TIMESTAMP 1 SET c = ?, val = ? WHERE k = 
> ?", 0, 0, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET c = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s USING TIMESTAMP 2 SET val = ? WHERE k = ?", 1, 
> 0);
> updateView("UPDATE %s USING TIMESTAMP 4 SET c = ? WHERE k = ?", 2, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET val = ? WHERE k = ?", 2, 
> 0);
> // FIXME no rows return
> assertRows(execute("SELECT c, k, val FROM mv_rctstest limit 1"), 
> row(2, 0, 2));
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(2, 0, 
> 2));
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13883) StrictLiveness for view row is not handled in AbstractRow

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172606#comment-16172606
 ] 

ZhaoYang commented on CASSANDRA-13883:
--

| source | unit | dtest |
| 
[trunk|https://github.com/apache/cassandra/compare/trunk...jasonstack:CASSANDRA-13883-trunk?expand=1]|
 [passed|https://circleci.com/gh/jasonstack/cassandra/627] |   |
| 
[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...jasonstack:CASSANDRA-13883-3.11?expand=1]
 | [passed|https://circleci.com/gh/jasonstack/cassandra/625] |  |
| 
[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...jasonstack:CASSANDRA-13883-3.0?expand=1]
 | |  |
| 
[dtest|https://github.com/apache/cassandra-dtest/compare/master...jasonstack:CASSANDRA-13883?expand=1]
 |

CI looks good, I will restart a few flaky ones.

{code}
Changes:
1. Change {{AbstractRow.hasLiveData}} to check {{enforceStrictLiveness}}:  
if livenessInfo is not live and enforceStrictLiveness, then there is not live 
data.
2. For SPRC.group, use the first command to get {{enforceStrictLiveness}} 
since each command should be the same except for key.
{code}

> StrictLiveness for view row is not handled in AbstractRow
> -
>
> Key: CASSANDRA-13883
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13883
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> In {{AbstractRow.hasLiveData(nowInSecond)}}, it doesn't handle 
> {{strictLiveness}} introduced in CASSANDRA-11500. The {{DataLimits}} counts 
> the expired view row as live data and then the expired view row is purged in 
> {{Row.purge()}}. When query with limit, we will get less data.
> {code:title=test to reproduce}
> @Test
> public void testRegularColumnTimestampUpdates() throws Throwable
> {
> createTable("CREATE TABLE %s (" +
> "k int PRIMARY KEY, " +
> "c int, " +
> "val int)");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("mv_rctstest", "CREATE MATERIALIZED VIEW %s AS SELECT * 
> FROM %%s WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)");
> updateView("UPDATE %s SET c = ?, val = ? WHERE k = ?", 0, 0, 0);
> updateView("UPDATE %s SET val = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s SET c = ? WHERE k = ?", 1, 0);
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(1, 0, 
> 1));
> updateView("TRUNCATE %s");
> updateView("UPDATE %s USING TIMESTAMP 1 SET c = ?, val = ? WHERE k = 
> ?", 0, 0, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET c = ? WHERE k = ?", 1, 0);
> updateView("UPDATE %s USING TIMESTAMP 2 SET val = ? WHERE k = ?", 1, 
> 0);
> updateView("UPDATE %s USING TIMESTAMP 4 SET c = ? WHERE k = ?", 2, 0);
> updateView("UPDATE %s USING TIMESTAMP 3 SET val = ? WHERE k = ?", 2, 
> 0);
> // FIXME no rows return
> assertRows(execute("SELECT c, k, val FROM mv_rctstest limit 1"), 
> row(2, 0, 2));
> assertRows(execute("SELECT c, k, val FROM mv_rctstest"), row(2, 0, 
> 2));
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13747) Fix AssertionError in short read protection

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13747:
---
Labels: Correctness  (was: )

> Fix AssertionError in short read protection
> ---
>
> Key: CASSANDRA-13747
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13747
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>  Labels: Correctness
> Fix For: 3.0.15, 3.11.1
>
>
> {{ShortReadRowProtection.moreContents()}} expects that by the time we get to 
> that method, the global post-reconciliation counter was already applied to 
> the current partition. However, sometimes it won’t get applied, and the 
> global counter continues counting with {{rowInCurrentPartition}} value not 
> reset from previous partition, which in the most obvious case would trigger 
> the assertion we are observing - {{assert 
> !postReconciliationCounter.isDoneForPartition();}}. In other cases it’s 
> possible because of this lack of reset to query a node for too few extra 
> rows, causing unnecessary SRP data requests.
> Why is the counter not always applied to the current partition?
> The merged {{PartitionIterator}} returned from {{DataResolver.resolve()}} has 
> two transformations applied to it, in the following order:
> {{Filter}} - to purge non-live data from partitions, and to discard empty 
> partitions altogether (except for Thrift)
> {{Counter}}, to count and stop iteration
> Problem is, {{Filter}} ’s {{applyToPartition()}} code that discards empty 
> partitions ({{closeIfEmpty()}} method) would sometimes consume the iterator, 
> triggering short read protection *before* {{Counter}} ’s 
> {{applyToPartition()}} gets called and resets its {{rowInCurrentPartition}} 
> sub-counter.
> We should not be consuming iterators until all transformations are applied to 
> them. For transformations it means that they cannot consume iterators unless 
> they are the last transformation on the stack.
> The linked branch fixes the problem by splitting {{Filter}} into two 
> transformations. The original - {{Filter}} - that does filtering within 
> partitions - and a separate {{EmptyPartitionsDiscarder}}, that discards empty 
> partitions from {{PartitionIterators}}. Thus {{DataResolve.resolve()}}, when 
> constructing its {{PartitionIterator}}, now does merge first, then applies 
> {{Filter}}, then {{Counter}}, and only then, as its last (third) 
> transformation - the {{EmptyPartitionsDiscarder}}. Being the last one 
> applied, it’s legal for it to consume the iterator, and triggering 
> {{moreContents()}} is now no longer a problem.
> Fixes: [3.0|https://github.com/iamaleksey/cassandra/commits/13747-3.0], 
> [3.11|https://github.com/iamaleksey/cassandra/commits/13747-3.11], 
> [4.0|https://github.com/iamaleksey/cassandra/commits/13747-4.0]. dtest 
> [here|https://github.com/iamaleksey/cassandra-dtest/commits/13747].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13404) Hostname verification for client-to-node encryption

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172394#comment-16172394
 ] 

Jason Brown commented on CASSANDRA-13404:
-

In client-side endpoint verification, the client knows the server's hostname, 
retrieves the server's cert, and compares the hostname it already has with 
what's in the cert. In other words, it uses a piece of information it already 
had as well as a piece of information from the server.

In the case of server-side enpoint verification, what you are proposing is to 
use the client IP address from the incoming socket as the hostname and then a 
cert that will also be provided by the client. Thus, both pieces of information 
that the server uses to verify are coming from the client. The certificate is 
cryptographically signed, but the IP address is not. This means that the IP 
address could potentially be spoofed and will not satisfy the original 
requirement of "preventing certificates from being copied and used elsewhere". 

bq. Hostname verification on the server side would improve security in that no 
one is able to copy a certificate from a client node to another location, and 
then use it to manipulate data in Cassandra.

By the time an attacker can copy a cert, can't they also spoof an IP address, 
as well?

I'll be honest, I'm unconfortable with the patch - taking the incoming IP 
address and passing that directly into the {{SslHandler}} just seems wrong. 
More importantly, I don't think this solution really solves your problem of 
ensuring that only legit clients can connect as IP addresses can be spoofed.

I'm -1 unless [~spo...@gmail.com] or anybody else has a convincing argument.

/cc [~smitchelus] of the core netty team, as he and i discussed this issue at 
length.


> Hostname verification for client-to-node encryption
> ---
>
> Key: CASSANDRA-13404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13404-trunk.txt
>
>
> Similarily to CASSANDRA-9220, Cassandra should support hostname verification 
> for client-node connections.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172160#comment-16172160
 ] 

Ariel Weisberg commented on CASSANDRA-13862:


I did 3 dtest runs and compared the results to trunk and they looked correct. 
I'll commit when I get a chance.

> Optimize Paxos prepare and propose stage for local requests 
> 
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire 
> MessagingService stack in Cassandra even if request is to be served locally, 
> we can enhance and make local requests severed w/o involving 
> MessagingService. Similar things are done at may [other places | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
>  in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra 
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D 
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 |  A.B.C.D 
> |  15045
> … 
>  
> REQUEST_RESPONSE message received from /A.B.C.D 
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 |  A.B.C.D 
> |  20270
> … 
>   
>  Processing response from /A.B.C.D [SharedPool-Worker-4] 
> | 2017-09-11 21:55:18.976000 |  A.B.C.D |  20372
> {quote}
> Same thing applies for {{Propose stage}} as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-13862:
---
Status: Ready to Commit  (was: Patch Available)

> Optimize Paxos prepare and propose stage for local requests 
> 
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire 
> MessagingService stack in Cassandra even if request is to be served locally, 
> we can enhance and make local requests severed w/o involving 
> MessagingService. Similar things are done at may [other places | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
>  in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra 
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D 
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 |  A.B.C.D 
> |  15045
> … 
>  
> REQUEST_RESPONSE message received from /A.B.C.D 
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 |  A.B.C.D 
> |  20270
> … 
>   
>  Processing response from /A.B.C.D [SharedPool-Worker-4] 
> | 2017-09-11 21:55:18.976000 |  A.B.C.D |  20372
> {quote}
> Same thing applies for {{Propose stage}} as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172137#comment-16172137
 ] 

Aleksey Yeschenko commented on CASSANDRA-12373:
---

It's a case of better late than never. It's also a necessary JIRA to allow to 
move to 4.0.

> 3.0 breaks CQL compatibility with super columns families
> 
>
> Key: CASSANDRA-12373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12373
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Sylvain Lebresne
>Assignee: Alex Petrov
> Fix For: 3.0.x, 3.11.x
>
>
> This is a follow-up to CASSANDRA-12335 to fix the CQL side of super column 
> compatibility.
> The details and a proposed solution can be found in the comments of 
> CASSANDRA-12335 but the crux of the issue is that super column famillies show 
> up differently in CQL in 3.0.x/3.x compared to 2.x, hence breaking backward 
> compatibilty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families

[sankalp kohli (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172132#comment-16172132
 ] 

sankalp kohli commented on CASSANDRA-12373:
---

Is this not too late for 3.0? 

> 3.0 breaks CQL compatibility with super columns families
> 
>
> Key: CASSANDRA-12373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12373
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Sylvain Lebresne
>Assignee: Alex Petrov
> Fix For: 3.0.x, 3.11.x
>
>
> This is a follow-up to CASSANDRA-12335 to fix the CQL side of super column 
> compatibility.
> The details and a proposed solution can be found in the comments of 
> CASSANDRA-12335 but the crux of the issue is that super column famillies show 
> up differently in CQL in 3.0.x/3.x compared to 2.x, hence breaking backward 
> compatibilty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13884) sstableloader option to accept target keyspace name

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia reassigned CASSANDRA-13884:
-

Assignee: Jaydeepkumar Chovatia

> sstableloader option to accept target keyspace name
> ---
>
> Key: CASSANDRA-13884
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13884
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Often as part of backup people store entire {{data}} directory. When they see 
> some corruption in data then they would like to restore data in same cluster 
> (for large clusters 200 nodes) but with different keyspace name. 
> Currently {{sstableloader}} uses parent folder as {{keyspace}}, it would be 
> nice to have an option to specify target keyspace name as part of 
> {{sstableloader}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13816) Document and test CAS and non-CAS batch behavior for deleting and inserting the same key

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172029#comment-16172029
 ] 

Michael Shuler commented on CASSANDRA-13816:


(removed 3.x fixver - there are no more 3.x releases)

> Document and test CAS and non-CAS batch behavior for deleting and inserting 
> the same key
> 
>
> Key: CASSANDRA-13816
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13816
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website, Testing
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Minor
> Fix For: 3.11.x, 4.x
>
>
> Add/verify unit tests for inserting and deleting the same key for cell 
> deletion, partition deletion, and range tombstone deletion for both CAS and 
> non-CAS batches.
> Don't change the existing behavior.
> The behavior differs between batch and CAS so in the both the CAS and batch 
> documentation mention that the behavior is not consistent. Make sure it is 
> visible in both high level and reference docs for that functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13816) Document and test CAS and non-CAS batch behavior for deleting and inserting the same key

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13816:
---
Fix Version/s: (was: 3.x)

> Document and test CAS and non-CAS batch behavior for deleting and inserting 
> the same key
> 
>
> Key: CASSANDRA-13816
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13816
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website, Testing
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Minor
> Fix For: 3.11.x, 4.x
>
>
> Add/verify unit tests for inserting and deleting the same key for cell 
> deletion, partition deletion, and range tombstone deletion for both CAS and 
> non-CAS batches.
> Don't change the existing behavior.
> The behavior differs between batch and CAS so in the both the CAS and batch 
> documentation mention that the behavior is not consistent. Make sure it is 
> visible in both high level and reference docs for that functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13872) document speculative_retry on DDL page

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13872:
---
Fix Version/s: 4.x
   3.11.x

> document speculative_retry on DDL page
> --
>
> Key: CASSANDRA-13872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13872
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Jon Haddad
>  Labels: lhf
> Fix For: 3.11.x, 4.x
>
>
> There's no mention of speculative_retry or how it works on 
> https://cassandra.apache.org/doc/latest/cql/ddl.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13872) document speculative_retry on DDL page

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13872:
---
Fix Version/s: (was: 3.11.1)
   (was: 4.0)

> document speculative_retry on DDL page
> --
>
> Key: CASSANDRA-13872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13872
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Jon Haddad
>  Labels: lhf
> Fix For: 3.11.x, 4.x
>
>
> There's no mention of speculative_retry or how it works on 
> https://cassandra.apache.org/doc/latest/cql/ddl.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13872) document speculative_retry on DDL page

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13872:
---
Issue Type: Improvement  (was: Bug)

> document speculative_retry on DDL page
> --
>
> Key: CASSANDRA-13872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13872
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Jon Haddad
>  Labels: lhf
> Fix For: 3.11.1, 4.0
>
>
> There's no mention of speculative_retry or how it works on 
> https://cassandra.apache.org/doc/latest/cql/ddl.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13661) java.lang.RuntimeException: 30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not defined as a collection

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13661:
---
Priority: Major  (was: Critical)

> java.lang.RuntimeException: 
> 30623431613136352d656433372d343939322d393066342d366632313961393530353062 is 
> not defined as a collection
> ---
>
> Key: CASSANDRA-13661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13661
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Rekha Joshi
>
> ERROR [CompactionExecutor:9] 2017-07-03 21:20:31,221 CassandraDaemon.java:231 
> - Exception in thread Thread[CompactionExecutor:9,1,main]
> java.lang.RuntimeException: 
> 30623431613136352d656433372d343939322d393066342d366632313961393530353062 is 
> not defined as a collection



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13556) Corrupted SSTables

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13556:
---
Priority: Major  (was: Critical)

> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
> Fix For: 3.11.x
>
>
> After 3 month of working, we noticed that number of compaction tasks were 
> growing (~600 pending tasks). SStables verification shows that some of them 
> were corrupted. Repairing didn't help (it was crashing with error). 
> Also some of requests (f.e. select * from fetcher where 
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
> error:
> {color:red}
> Traceback (most recent call last):
>   File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
> perform_simple_statement
> result = future.result()
>   File 
> "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
>  line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
>   'sstable_size_in_mb': '256',
>   'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
>'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted 
> [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13556) Corrupted SSTables

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13556:
---
Fix Version/s: (was: 3.9)
   3.11.x

> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
>Priority: Critical
> Fix For: 3.11.x
>
>
> After 3 month of working, we noticed that number of compaction tasks were 
> growing (~600 pending tasks). SStables verification shows that some of them 
> were corrupted. Repairing didn't help (it was crashing with error). 
> Also some of requests (f.e. select * from fetcher where 
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
> error:
> {color:red}
> Traceback (most recent call last):
>   File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
> perform_simple_statement
> result = future.result()
>   File 
> "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
>  line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
>   'sstable_size_in_mb': '256',
>   'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
>'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted 
> [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13591:
---
Priority: Major  (was: Critical)

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better placed at [https://github.com/lz4/lz4-java] but i'm 
> not sure how to get better details on the error.
> Potentially due to data corruption.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13669) Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13669:
---
Reproduced In: 3.11.0, 3.11.1, 4.0

> Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
> -
>
> Key: CASSANDRA-13669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13669
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Tested on:
> * macOS Sierra 10.12.5
> * Ubuntu 14.04.5 LTS
>Reporter: Lukasz Biedrycki
>Priority: Critical
>  Labels: sasi
> Fix For: 3.11.x
>
>
> Recently I experienced a problem that prevents me to restart cassandra.
> I narrowed it down to SASI Index when added on uuid field.
> Steps to reproduce:
> 1. start cassandra (./bin/cassandra -f)
> 2. create keyspace, table, index and add data:
> {noformat}
> CREATE KEYSPACE testkeyspace
> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} 
>AND durable_writes = true;
> use testkeyspace ;
> CREATE TABLE testtable (
>col1 uuid,
>col2 uuid,
>ts timeuuid,
>col3 uuid,
>PRIMARY KEY((col1, col2), ts) ) with clustering order by (ts desc);
> CREATE CUSTOM INDEX col3_testtable_idx ON testtable(col3)
> USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'mode': 
> 'PREFIX'};
> INSERT INTO testtable(col1, col2, ts, col3)
> VALUES(898e0014-6161-11e7-b9b7-238ea83bd70b,
>898e0014-6161-11e7-b9b7-238ea83bd70b,
>now(), 898e0014-6161-11e7-b9b7-238ea83bd70b);
> {noformat}
> 3. restart cassandra
> It crashes with an error (sorry it's huge):
> {noformat}
> DEBUG 09:09:20 Writing Memtable-testtable@1005362073(0.075KiB serialized 
> bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = 
> (min(-9223372036854775808), max(9223372036854775807)]
> ERROR 09:09:20 Exception in thread 
> Thread[PerDiskMemtableFlushWriter_0:1,5,main]
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
>   at 
> org.apache.cassandra.db.marshal.UUIDType.fromString(UUIDType.java:118) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer.hasNext(StandardAnalyzer.java:168)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.add(PerSSTableIndexWriter.java:208)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.lambda$nextUnfilteredCluster$0(PerSSTableIndexWriter.java:132)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.Collections$SingletonSet.forEach(Collections.java:4767) 
> ~[na:1.8.0_131]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.nextUnfilteredCluster(PerSSTableIndexWriter.java:119)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.lambda$add$1(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131]
>   at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:107) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> 

[jira] [Commented] (CASSANDRA-13669) Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171998#comment-16171998
 ] 

Michael Shuler commented on CASSANDRA-13669:


Reproduced on 3.11.0 release sha, cassandra-3.11 branch HEAD (commit 594f1c1d), 
and trunk branch HEAD (commit eb76692).

> Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
> -
>
> Key: CASSANDRA-13669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13669
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Tested on:
> * macOS Sierra 10.12.5
> * Ubuntu 14.04.5 LTS
>Reporter: Lukasz Biedrycki
>Priority: Critical
>  Labels: sasi
> Fix For: 3.11.x
>
>
> Recently I experienced a problem that prevents me to restart cassandra.
> I narrowed it down to SASI Index when added on uuid field.
> Steps to reproduce:
> 1. start cassandra (./bin/cassandra -f)
> 2. create keyspace, table, index and add data:
> {noformat}
> CREATE KEYSPACE testkeyspace
> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} 
>AND durable_writes = true;
> use testkeyspace ;
> CREATE TABLE testtable (
>col1 uuid,
>col2 uuid,
>ts timeuuid,
>col3 uuid,
>PRIMARY KEY((col1, col2), ts) ) with clustering order by (ts desc);
> CREATE CUSTOM INDEX col3_testtable_idx ON testtable(col3)
> USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'mode': 
> 'PREFIX'};
> INSERT INTO testtable(col1, col2, ts, col3)
> VALUES(898e0014-6161-11e7-b9b7-238ea83bd70b,
>898e0014-6161-11e7-b9b7-238ea83bd70b,
>now(), 898e0014-6161-11e7-b9b7-238ea83bd70b);
> {noformat}
> 3. restart cassandra
> It crashes with an error (sorry it's huge):
> {noformat}
> DEBUG 09:09:20 Writing Memtable-testtable@1005362073(0.075KiB serialized 
> bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = 
> (min(-9223372036854775808), max(9223372036854775807)]
> ERROR 09:09:20 Exception in thread 
> Thread[PerDiskMemtableFlushWriter_0:1,5,main]
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
>   at 
> org.apache.cassandra.db.marshal.UUIDType.fromString(UUIDType.java:118) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer.hasNext(StandardAnalyzer.java:168)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.add(PerSSTableIndexWriter.java:208)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.lambda$nextUnfilteredCluster$0(PerSSTableIndexWriter.java:132)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.Collections$SingletonSet.forEach(Collections.java:4767) 
> ~[na:1.8.0_131]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.nextUnfilteredCluster(PerSSTableIndexWriter.java:119)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.lambda$add$1(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131]
>   at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:107) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> org.apache.cassandra.serializers.MarshalException: Unable 

[jira] [Commented] (CASSANDRA-13530) GroupCommitLogService

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171928#comment-16171928
 ] 

Ariel Weisberg commented on CASSANDRA-13530:


I've run into a different justification for this change. Reducing P/E cycles on 
SSDs that don't have a non-volatile write cache sitting in front.

So we don't need to justify having this functionality with just performance and 
batch would still do the wrong thing even if it was faster.

Just a heads up this can only go into trunk. 2.2,  3.0, and 3.11 are bug fix 
only. So in 2.2, 3.0 and 3.11 we could fix batch by ignoring the value 
commitlog_sync_batch_window_in_ms and updating the documentation in 
cassandra.yaml as well as mentioning that it's deprecated.

So on to reviewing the details.

* This needs a unit test exercising the new commit log configuration, as well 
as acceptable and not acceptable configuration values (0, negative, largest 
value, smallest value)
* How about we fix commitlog_sync_batch_window_in_ms by 100% ignoring the 
value. We can't change the configuration behavior of batch (people won't update 
their config) but we can fix the broken implementation. Also mark it as 
deprecated in the docs so we can remove it later.
* Then use commitlog_sync_group_window_in_ms to configure the batch commit log 
service. If unset, batch works the way it does today. If set then it implements 
the fixed grouping behavior. Document in cassandra.yaml appropriately.
* Alternatively add a new commit log implementation (group) that can do both 
group and batch and deprecate batch commit log. Eventually we will converge on 
one implementation with consistent configuration naming.


> GroupCommitLogService
> -
>
> Key: CASSANDRA-13530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13530
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Yuji Ito
>Assignee: Yuji Ito
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
> Attachments: groupAndBatch.png, groupCommit22.patch, 
> groupCommit30.patch, groupCommit3x.patch, 
> groupCommitLog_noSerial_result.xlsx, groupCommitLog_result.xlsx, 
> GuavaRequestThread.java, MicroRequestThread.java
>
>
> I propose a new CommitLogService, GroupCommitLogService, to improve the 
> throughput when lots of requests are received.
> It improved the throughput by maximum 94%.
> I'd like to discuss about this CommitLogService.
> Currently, we can select either 2 CommitLog services; Periodic and Batch.
> In Periodic, we might lose some commit log which hasn't written to the disk.
> In Batch, we can write commit log to the disk every time. The size of commit 
> log to write is too small (< 4KB). When high concurrency, these writes are 
> gathered and persisted to the disk at once. But, when insufficient 
> concurrency, many small writes are issued and the performance decreases due 
> to the latency of the disk. Even if you use SSD, processes of many IO 
> commands decrease the performance.
> GroupCommitLogService writes some commitlog to the disk at once.
> The patch adds GroupCommitLogService (It is enabled by setting 
> `commitlog_sync` and `commitlog_sync_group_window_in_ms` in cassandra.yaml).
> The difference from Batch is just only waiting for the semaphore.
> By waiting for the semaphore, some writes for commit logs are executed at the 
> same time.
> In GroupCommitLogService, the latency becomes worse if the there is no 
> concurrency.
> I measured the performance with my microbench (MicroRequestThread.java) by 
> increasing the number of threads.The cluster has 3 nodes (Replication factor: 
> 3). Each nodes is AWS EC2 m4.large instance + 200IOPS io1 volume.
> The result is as below. The GroupCommitLogService with 10ms window improved 
> update with Paxos by 94% and improved select with Paxos by 76%.
> h6. SELECT / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|192|103|
> |2|163|212|
> |4|264|416|
> |8|454|800|
> |16|744|1311|
> |32|1151|1481|
> |64|1767|1844|
> |128|2949|3011|
> |256|4723|5000|
> h6. UPDATE / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|45|26|
> |2|39|51|
> |4|58|102|
> |8|102|198|
> |16|167|213|
> |32|289|295|
> |64|544|548|
> |128|1046|1058|
> |256|2020|2061|



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13631) add header parameter size to internode messaging protocol

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171906#comment-16171906
 ] 

Jason Brown edited comment on CASSANDRA-13631 at 9/19/17 3:32 PM:
--

Here is a patch that repurposes the 4-byte parameters count to the total count 
of bytes in the parameters.

||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|



was (Author: jasobrown):
||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|


> add header parameter size to internode messaging protocol
> -
>
> Key: CASSANDRA-13631
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13631
> Project: Cassandra
>  Issue Type: Task
>  Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> as netty is not a streaming/blocking protocol and works best when buffer 
> sizes are known, I had to do a bunch of contortions in {{MessageInHandler}} 
> (as part of CASSANDRA-8457) to safely read the message header parameters. If 
> we add a header parameters size field to the internode messaging protocol, 
> the header parsing code would be dramatically simpler (note: we'll still need 
> the existing parsing to support the cluster upgrade use case). An alternative 
> to adding a new field is to hijack the existing header parameter count field; 
> that field is an {{int}}, so field width is not an issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13631) add header parameter size to internode messaging protocol

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13631:

Fix Version/s: 4.0
   Status: Patch Available  (was: Open)

||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|


> add header parameter size to internode messaging protocol
> -
>
> Key: CASSANDRA-13631
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13631
> Project: Cassandra
>  Issue Type: Task
>  Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> as netty is not a streaming/blocking protocol and works best when buffer 
> sizes are known, I had to do a bunch of contortions in {{MessageInHandler}} 
> (as part of CASSANDRA-8457) to safely read the message header parameters. If 
> we add a header parameters size field to the internode messaging protocol, 
> the header parsing code would be dramatically simpler (note: we'll still need 
> the existing parsing to support the cluster upgrade use case). An alternative 
> to adding a new field is to hijack the existing header parameter count field; 
> that field is an {{int}}, so field width is not an issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13669) Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13669:

Labels: sasi  (was: )

> Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
> -
>
> Key: CASSANDRA-13669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13669
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Tested on:
> * macOS Sierra 10.12.5
> * Ubuntu 14.04.5 LTS
>Reporter: Lukasz Biedrycki
>Priority: Critical
>  Labels: sasi
> Fix For: 3.11.x
>
>
> Recently I experienced a problem that prevents me to restart cassandra.
> I narrowed it down to SASI Index when added on uuid field.
> Steps to reproduce:
> 1. start cassandra (./bin/cassandra -f)
> 2. create keyspace, table, index and add data:
> {noformat}
> CREATE KEYSPACE testkeyspace
> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} 
>AND durable_writes = true;
> use testkeyspace ;
> CREATE TABLE testtable (
>col1 uuid,
>col2 uuid,
>ts timeuuid,
>col3 uuid,
>PRIMARY KEY((col1, col2), ts) ) with clustering order by (ts desc);
> CREATE CUSTOM INDEX col3_testtable_idx ON testtable(col3)
> USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'mode': 
> 'PREFIX'};
> INSERT INTO testtable(col1, col2, ts, col3)
> VALUES(898e0014-6161-11e7-b9b7-238ea83bd70b,
>898e0014-6161-11e7-b9b7-238ea83bd70b,
>now(), 898e0014-6161-11e7-b9b7-238ea83bd70b);
> {noformat}
> 3. restart cassandra
> It crashes with an error (sorry it's huge):
> {noformat}
> DEBUG 09:09:20 Writing Memtable-testtable@1005362073(0.075KiB serialized 
> bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = 
> (min(-9223372036854775808), max(9223372036854775807)]
> ERROR 09:09:20 Exception in thread 
> Thread[PerDiskMemtableFlushWriter_0:1,5,main]
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
>   at 
> org.apache.cassandra.db.marshal.UUIDType.fromString(UUIDType.java:118) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer.hasNext(StandardAnalyzer.java:168)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.add(PerSSTableIndexWriter.java:208)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.lambda$nextUnfilteredCluster$0(PerSSTableIndexWriter.java:132)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.Collections$SingletonSet.forEach(Collections.java:4767) 
> ~[na:1.8.0_131]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.nextUnfilteredCluster(PerSSTableIndexWriter.java:119)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.lambda$add$1(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131]
>   at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:107) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.lang.RuntimeException: 

[jira] [Updated] (CASSANDRA-13669) Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13669:
---
Fix Version/s: (was: 3.11.0)
   (was: 3.9)
   3.11.x

> Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
> -
>
> Key: CASSANDRA-13669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13669
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Tested on:
> * macOS Sierra 10.12.5
> * Ubuntu 14.04.5 LTS
>Reporter: Lukasz Biedrycki
>Priority: Critical
> Fix For: 3.11.x
>
>
> Recently I experienced a problem that prevents me to restart cassandra.
> I narrowed it down to SASI Index when added on uuid field.
> Steps to reproduce:
> 1. start cassandra (./bin/cassandra -f)
> 2. create keyspace, table, index and add data:
> {noformat}
> CREATE KEYSPACE testkeyspace
> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} 
>AND durable_writes = true;
> use testkeyspace ;
> CREATE TABLE testtable (
>col1 uuid,
>col2 uuid,
>ts timeuuid,
>col3 uuid,
>PRIMARY KEY((col1, col2), ts) ) with clustering order by (ts desc);
> CREATE CUSTOM INDEX col3_testtable_idx ON testtable(col3)
> USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'mode': 
> 'PREFIX'};
> INSERT INTO testtable(col1, col2, ts, col3)
> VALUES(898e0014-6161-11e7-b9b7-238ea83bd70b,
>898e0014-6161-11e7-b9b7-238ea83bd70b,
>now(), 898e0014-6161-11e7-b9b7-238ea83bd70b);
> {noformat}
> 3. restart cassandra
> It crashes with an error (sorry it's huge):
> {noformat}
> DEBUG 09:09:20 Writing Memtable-testtable@1005362073(0.075KiB serialized 
> bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = 
> (min(-9223372036854775808), max(9223372036854775807)]
> ERROR 09:09:20 Exception in thread 
> Thread[PerDiskMemtableFlushWriter_0:1,5,main]
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
>   at 
> org.apache.cassandra.db.marshal.UUIDType.fromString(UUIDType.java:118) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer.hasNext(StandardAnalyzer.java:168)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.add(PerSSTableIndexWriter.java:208)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.lambda$nextUnfilteredCluster$0(PerSSTableIndexWriter.java:132)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.Collections$SingletonSet.forEach(Collections.java:4767) 
> ~[na:1.8.0_131]
>   at 
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.nextUnfilteredCluster(PerSSTableIndexWriter.java:119)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.lambda$add$1(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131]
>   at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:233) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:107) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from 
> 'aa'
> java.lang.RuntimeException: 

[jira] [Commented] (CASSANDRA-13757) Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171873#comment-16171873
 ] 

Michael Shuler commented on CASSANDRA-13757:


3.5 has a lot of bugs fixed in subsequent versions up through 3.11.0, so I'd be 
interested to know if this reproduces in 3.11.0. I'm also working on 3.11.1 
release soon, if you want to give a try after that has been released.

> Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
> 
>
> Key: CASSANDRA-13757
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13757
> Project: Cassandra
>  Issue Type: Bug
> Environment: Operation System: Debian Jessie
> Java: Oracle JDK 1.8.0_131
> Cassandra: 3.5.0
>Reporter: Serhat Rıfat Demircan
>
> We got following error while repair job running on our cluster. One of the 
> nodes stop due to segmantation fault in JVM and repair job fails.
> We could not reproduce this problem on our test and staging enviroment (main 
> difference is data size).
> {code:java}
> #
> #  SIGSEGV (0xb) at pc=0x7fd80a399e70, pid=1305, tid=0x7fd7ee7c4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 
> 1.8.0_131-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> ---  T H R E A D  ---
> Current thread (0x7fce32dad1b0):  JavaThread "CompactionExecutor:9798" 
> daemon [_thread_in_native, id=16879, 
> stack(0x7fd7ee784000,0x7fd7ee7c5000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
> 0x7fd450c4d000
> Registers:
> RAX=0x7fcde6560d32, RBX=0x7fd450c4cff9, RCX=0x7fcde6560c7a, 
> RDX=0x7fcde6560d3e
> RSP=0x7fd7ee7c3160, RBP=0x7fd450c44ae6, RSI=0x7fcde6562ff8, 
> RDI=0x00c2
> R8 =0x7fcde6562ff4, R9 =0x7fcde6563000, R10=0x, 
> R11=0x
> R12=0x000c, R13=0x7fd4501cd000, R14=0x7fcde6562ff7, 
> R15=0x7fcde6562ffb
> RIP=0x7fd80a399e70, EFLAGS=0x00010283, CSGSFS=0x0033, 
> ERR=0x0004
>   TRAPNO=0x000e
> Top of Stack: (sp=0x7fd7ee7c3160)
> 0x7fd7ee7c3160:   0008 7fd81e21c3d0
> 0x7fd7ee7c3170:   0004 0001
> 0x7fd7ee7c3180:   0002 0001
> 0x7fd7ee7c3190:   0004 0004
> 0x7fd7ee7c31a0:   0004 0004
> 0x7fd7ee7c31b0:    
> 0x7fd7ee7c31c0:    
> 0x7fd7ee7c31d0:    0001
> 0x7fd7ee7c31e0:   0002 0003
> 0x7fd7ee7c31f0:   7fd7ee7c32b8 7fce32dad3a8
> 0x7fd7ee7c3200:    
> 0x7fd7ee7c3210:   7fd4501cd000 7fcde6553000
> 0x7fd7ee7c3220:   00a77ae6 7fd80a39659d
> 0x7fd7ee7c3230:    dcb8fc9b
> 0x7fd7ee7c3240:   7fd7ee7c32d0 
> 0x7fd7ee7c3250:   0006e5c7e4d8 7fd7ee7c32b8
> 0x7fd7ee7c3260:   7fce32dad1b0 7fd81df2099d
> 0x7fd7ee7c3270:   7fd7ee7c32a8 
> 0x7fd7ee7c3280:   0001 
> 0x7fd7ee7c3290:   0006e5c7e528 7fd81d74df10
> 0x7fd7ee7c32a0:    0006e5c7e4d8
> 0x7fd7ee7c32b0:   0006f6c7fbf8 0006f6e957f0
> 0x7fd7ee7c32c0:   0006e5c7e350 7fd87fff
> 0x7fd7ee7c32d0:   0006e5c7e528 7fd81fa867e0
> 0x7fd7ee7c32e0:   00a77ae20001 00a77ae2
> 0x7fd7ee7c32f0:   0006e5c7e488 0112d5f1
> 0x7fd7ee7c3300:   dcb8fc9b99ce 000100a77ae6
> 0x7fd7ee7c3310:   00a814b000a814b4 0006e5c7e4d8
> 0x7fd7ee7c3320:   0006e5c7e4d8 0006f6a4df38
> 0x7fd7ee7c3330:   00060001 00067fff
> 0x7fd7ee7c3340:   008971582c8a 0006189d87852057
> 0x7fd7ee7c3350:    e5244e71
> Instructions: (pc=0x7fd80a399e70)
> 0x7fd80a399e50:   e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20
> 0x7fd80a399e60:   48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00
> 0x7fd80a399e70:   48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39
> 

[jira] [Updated] (CASSANDRA-13757) Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13757:
---
Priority: Major  (was: Critical)

> Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
> 
>
> Key: CASSANDRA-13757
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13757
> Project: Cassandra
>  Issue Type: Bug
> Environment: Operation System: Debian Jessie
> Java: Oracle JDK 1.8.0_131
> Cassandra: 3.5.0
>Reporter: Serhat Rıfat Demircan
>
> We got following error while repair job running on our cluster. One of the 
> nodes stop due to segmantation fault in JVM and repair job fails.
> We could not reproduce this problem on our test and staging enviroment (main 
> difference is data size).
> {code:java}
> #
> #  SIGSEGV (0xb) at pc=0x7fd80a399e70, pid=1305, tid=0x7fd7ee7c4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 
> 1.8.0_131-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> ---  T H R E A D  ---
> Current thread (0x7fce32dad1b0):  JavaThread "CompactionExecutor:9798" 
> daemon [_thread_in_native, id=16879, 
> stack(0x7fd7ee784000,0x7fd7ee7c5000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
> 0x7fd450c4d000
> Registers:
> RAX=0x7fcde6560d32, RBX=0x7fd450c4cff9, RCX=0x7fcde6560c7a, 
> RDX=0x7fcde6560d3e
> RSP=0x7fd7ee7c3160, RBP=0x7fd450c44ae6, RSI=0x7fcde6562ff8, 
> RDI=0x00c2
> R8 =0x7fcde6562ff4, R9 =0x7fcde6563000, R10=0x, 
> R11=0x
> R12=0x000c, R13=0x7fd4501cd000, R14=0x7fcde6562ff7, 
> R15=0x7fcde6562ffb
> RIP=0x7fd80a399e70, EFLAGS=0x00010283, CSGSFS=0x0033, 
> ERR=0x0004
>   TRAPNO=0x000e
> Top of Stack: (sp=0x7fd7ee7c3160)
> 0x7fd7ee7c3160:   0008 7fd81e21c3d0
> 0x7fd7ee7c3170:   0004 0001
> 0x7fd7ee7c3180:   0002 0001
> 0x7fd7ee7c3190:   0004 0004
> 0x7fd7ee7c31a0:   0004 0004
> 0x7fd7ee7c31b0:    
> 0x7fd7ee7c31c0:    
> 0x7fd7ee7c31d0:    0001
> 0x7fd7ee7c31e0:   0002 0003
> 0x7fd7ee7c31f0:   7fd7ee7c32b8 7fce32dad3a8
> 0x7fd7ee7c3200:    
> 0x7fd7ee7c3210:   7fd4501cd000 7fcde6553000
> 0x7fd7ee7c3220:   00a77ae6 7fd80a39659d
> 0x7fd7ee7c3230:    dcb8fc9b
> 0x7fd7ee7c3240:   7fd7ee7c32d0 
> 0x7fd7ee7c3250:   0006e5c7e4d8 7fd7ee7c32b8
> 0x7fd7ee7c3260:   7fce32dad1b0 7fd81df2099d
> 0x7fd7ee7c3270:   7fd7ee7c32a8 
> 0x7fd7ee7c3280:   0001 
> 0x7fd7ee7c3290:   0006e5c7e528 7fd81d74df10
> 0x7fd7ee7c32a0:    0006e5c7e4d8
> 0x7fd7ee7c32b0:   0006f6c7fbf8 0006f6e957f0
> 0x7fd7ee7c32c0:   0006e5c7e350 7fd87fff
> 0x7fd7ee7c32d0:   0006e5c7e528 7fd81fa867e0
> 0x7fd7ee7c32e0:   00a77ae20001 00a77ae2
> 0x7fd7ee7c32f0:   0006e5c7e488 0112d5f1
> 0x7fd7ee7c3300:   dcb8fc9b99ce 000100a77ae6
> 0x7fd7ee7c3310:   00a814b000a814b4 0006e5c7e4d8
> 0x7fd7ee7c3320:   0006e5c7e4d8 0006f6a4df38
> 0x7fd7ee7c3330:   00060001 00067fff
> 0x7fd7ee7c3340:   008971582c8a 0006189d87852057
> 0x7fd7ee7c3350:    e5244e71
> Instructions: (pc=0x7fd80a399e70)
> 0x7fd80a399e50:   e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20
> 0x7fd80a399e60:   48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00
> 0x7fd80a399e70:   48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39
> 0x7fd80a399e80:   c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48
> Register to memory mapping:
> RAX=0x7fcde6560d32 is an unknown value
> RBX=0x7fd450c4cff9 is an unknown value
> RCX=0x7fcde6560c7a is an unknown value
> 

[jira] [Resolved] (CASSANDRA-13792) unable to connect apache-cassandra-3.11.0 after place 4.2.2 JNA jar

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler resolved CASSANDRA-13792.

Resolution: Not A Problem

Closing as not a problem.
Both of the errors you posted appear to be basic linux cli or env problems.

{{./nodetool}} instead of {{nodetool}}
Install python >= 2.7.

> unable to connect apache-cassandra-3.11.0 after place 4.2.2 JNA jar 
> 
>
> Key: CASSANDRA-13792
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13792
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: Linux(RHEL 6.8)
>Reporter: rajasekhar
>Priority: Critical
> Fix For: 3.11.0
>
> Attachments: log_cassdb.txt
>
>
> After apply work around below solution, installled the pache-cassandra-3.11 
> but i am not connect server status and cqlsh. please find attached log for 
> more information  and let us know the instatllion completed successfully
> "Jeff Jirsa resolved CASSANDRA-13791.
> 
> Resolution: Duplicate
> This was caused (and resolved) by CASSANDRA-13072 , where JNA was upgraded to 
> {{4.4.0}}. That 4.4.0 JNA library was linked against a different version of 
> glibc ( {{GLIBC_2.14}} ), which causes this error. This is fixed in 
> {{3.11.1}}, but you can work around the issue by downloading the 4.2.2 JNA 
> jar from 
> [here|https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_raw_00a777ec8ab701b843172e23a6cbdc4d6cf48f8d_lib_jna-2D4.2.2.jar=DwICaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=zSydaPm_PKES-H2mB46-9X-iWMDIIymvQCfY02eRW-Q=cSmoTlLQJTt32Gb257uuaz6_WHE6UeXOV5XjO7GTcxk=yieYJ2V_ND3saGBjGH3O-DEwQdfkA44B9yBqtjIU8bE=
>  ] and placing it into the classpath ({{lib/}}), removing {{jna-4.4.0.jar}} 
> in the process.
> > unable to install apache-cassandra-3.11.0 in linux  box
> > ---
> "
> [cassdb@alsc_dev_db bin]$ nodetool status
> -bash: nodetool: command not found
> [cassdb@alsc_dev_db bin]$ pwd
> /u01/Cassandra_home/apache-cassandra-3.11.0/bin
> [cassdb@alsc_dev_db bin]$ sh 
> /u01/Cassandra_home/apache-cassandra-3.11.0/bin/cqlsh
> No appropriate python interpreter found.
> [cassdb@alsc_dev_db bin]$ service cassandra status



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13602) Trigger Data Unmarshalling not working for Compose Data Structures

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13602:
---
Priority: Major  (was: Blocker)

> Trigger Data Unmarshalling not working for Compose Data Structures
> --
>
> Key: CASSANDRA-13602
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13602
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Samuel Martinucci
>
> The following behavior was identified running Cassandra 3.10
> I have the following trigger:
> {code}
> public class TestTrigger implements ITrigger
> {
>   @Override
>   public Collection augment(final Partition update)
>   {
>   final UnfilteredRowIterator it = update.unfilteredIterator();
>   while (it.hasNext())
>   {
>   final Unfiltered un = it.next();
>   final Clustering clt = (Clustering) un.clustering();
>   final Row row = update.getRow(clt);
>   final Iterator cls = row.cells().iterator();
>   while (cls.hasNext())
>   {
>   final Cell cell = cls.next();
>   
> System.out.println(cell.column().name.toString() + " "
>   + 
> cell.column().cellValueType().getClass() + " "
>   + 
> cell.column().cellValueType().compose(cell.value()));
>   }
>   }
>   return Collections.emptyList();
>   }
> }
> {code}
> Configured for the following table:
> {code}
> CREATE TABLE IF NOT EXISTS audit_app_log
> (
>   id text,
>   log_time timestamp,
>   mappings map>,
>   PRIMARY KEY((id),log_time)
> )
> AND  CLUSTERING ORDER BY (log_time DESC);
> {code}
> When I insert data into this table, I have the following output:
> mappings class org.apache.cassandra.db.marshal.ListType ["test"]
> This output means that I am losing the key from the map, and seeing only the 
> internal data structure. Am I doing something wrong or this is really a bug?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12752) Exception encountered during startup (CFMetaData.rebuild failed)

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-12752:
---
Priority: Major  (was: Blocker)

> Exception encountered during startup (CFMetaData.rebuild failed)
> 
>
> Key: CASSANDRA-12752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12752
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.0.7, Debian Jessie. Java version 64 bits : 
> 1.8.0_91-b14, SSD software raid 1
>Reporter: Jérôme Schneider
>
> We have a cluster of 3 cassandra 3.0.7. We have the following problems :
> - ssltables was to big to be compress (not enough space left)
> - we launch a sstablesplit which seems to work
> - after we can't restart cassandra nodes we always have the same stacktrace  
> during the start :
> {quote}
> INFO  [main] 2016-10-05 18:02:47,346 ColumnFamilyStore.java:390 - 
> Initializing system_schema.aggregates
> INFO  [main] 2016-10-05 18:02:47,353 ColumnFamilyStore.java:390 - 
> Initializing system_schema.indexes
> ERROR [main] 2016-10-05 18:02:47,420 CassandraDaemon.java:698 - Exception 
> encountered during startup
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:313) 
> ~[apache-cassandra-3.0.7.jar:3.0.7]
> at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:292) 
> ~[apache-cassandra-3.0.7.jar:3.0.7]
> at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:368) 
> ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
>  ~[apache-cassandra-3.0.7.jar:3.0.7]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) 
> ~[apache-cassandra-3.0.7.jar:3.0.7]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) 
> ~[apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:235) 
> [apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>  [apache-cassandra-3.0.7.jar:3.0.7]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685) 
> [apache-cassandra-3.0.7.jar:3.0.7]
> {quote}
> - it's now impossible to start a cassandra node. I tried to : upgrade to 
> 3.0.9 (same error), I reset system_schema (cassandra start) and I tried to 
> launch sstablescrub on the corrupted keystore (same stacktrace).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13362) Cassandra 2.1.15 main thread stuck in logback stack trace upon joining existing cluster

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13362:
---
Priority: Major  (was: Blocker)

> Cassandra 2.1.15 main thread stuck in logback stack trace upon joining 
> existing cluster
> ---
>
> Key: CASSANDRA-13362
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13362
> Project: Cassandra
>  Issue Type: Bug
> Environment: 
>Reporter: Thomas Steinmaurer
> Attachments: td___2017-03-21-21-30-09.tdump, 
> td___2017-03-21-23-09-59.tdump
>
>
> Switching from Cassandra 2.0.17 to Cassandra 2.1.15 (DSC edition: 
> dsc-cassandra-2.1.15-bin.tar.gz) in a local VM based Linux environment for 
> installer verification tests.
> {noformat}
> [root@localhost jdk1.8.0_102]# lsb_release -d
> Description:  CentOS release 6.7 (Final)
> You have new mail in /var/spool/mail/root
> [root@localhost jdk1.8.0_102]# uname -a
> Linux localhost 2.6.32-573.el6.x86_64 #1 SMP Thu Jul 23 15:44:03 UTC 2015 
> x86_64 x86_64 x86_64 GNU/Linux
> {noformat}
> The test environment is started from scratch, thus in the following scenario 
> not an upgrade from 2.0 to 2.1, but a fresh 2.1 installation.
> The first node started up fine, but when extending the cluster with a second 
> node, the second node hangs in the following Cassandra log output while 
> starting up, joining the existing node:
> {noformat}
> INFO  [InternalResponseStage:1] 2017-03-21 21:10:43,864 DefsTables.java:373 - 
> Loading 
> org.apache.cassandra.config.CFMetaData@1c3daf27[cfId=a8cb1eb0-0e61-11e7-9a56-b20ca863,ksName=ruxitdb,cfName=EventQueue,cf$
> INFO  [main] 2017-03-21 21:11:11,404 StorageService.java:1138 - JOINING: 
> schema complete, ready to bootstrap
> ...
> INFO  [main] 2017-03-22 03:13:36,148 StorageService.java:1138 - JOINING: 
> waiting for pending range calculation
> INFO  [main] 2017-03-22 03:13:36,149 StorageService.java:1138 - JOINING: 
> calculation complete, ready to bootstrap
> INFO  [main] 2017-03-22 03:13:36,156 StorageService.java:1138 - JOINING: 
> getting bootstrap token
> ...
> {noformat}
> So, basically it was stuck on 2017-03-21 21:11:11,404 and the main thread 
> somehow continued on  2017-03-22 03:13:36,148, ~ 6 hours later.
> I have two thread dumps. The first from 21:30:
> [^td___2017-03-21-21-30-09.tdump]
> and a second one ~ 100min later:
> [^td___2017-03-21-23-09-59.tdump]
> Both thread dumps have in common, that the main thread is stuck in some 
> logback code:
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7fe93821a800 nid=0x4d4e waiting on 
> condition [0x7fe93c813000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0xc861bb88> (a 
> java.util.concurrent.locks.ReentrantLock$FairSync)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>   at 
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:224)
>   at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>   at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:217)
>   at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103)
>   at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
>   at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
>   at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273)
>   at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260)
>   at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442)
>   at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
>   at ch.qos.logback.classic.Logger.info(Logger.java:600)
>   at 
> org.apache.cassandra.service.StorageService.setMode(StorageService.java:1138)
>   at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:870)
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
>   - locked <0xc85d37d8> (a 
> org.apache.cassandra.service.StorageService)
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
>   - locked <0xc85d37d8> (a 
> org.apache.cassandra.service.StorageService)
>   at 

[jira] [Commented] (CASSANDRA-13885) Allow to run full repairs in 3.0 without additional cost of anti-compaction

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171831#comment-16171831
 ] 

Jeff Jirsa commented on CASSANDRA-13885:


cc [~krummas] and [~bdeggleston] for visibility.

It'll be far less invasive to remove (parts of?) CASSANDRA-7586 than it would 
be to backport CASSANDRA-9143 and the ~10 or so follow-up patches 
[~bdeggleston] has done to make incremental repair viable in trunk. 

Neither of these options feel very appropriate for 3.0 though, if I'm being 
honest.





> Allow to run full repairs in 3.0 without additional cost of anti-compaction
> ---
>
> Key: CASSANDRA-13885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13885
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Thomas Steinmaurer
>
> This ticket is basically the result of the discussion in Cassandra user list: 
> https://www.mail-archive.com/user@cassandra.apache.org/msg53562.html
> I was asked to open a ticket by Paulo Motta to think about back-porting 
> running full repairs without the additional cost of anti-compaction.
> Basically there is no way in 3.0 to run full repairs from several nodes 
> concurrently without troubles caused by (overlapping?) anti-compactions. 
> Coming from 2.1 this is a major change from an operational POV, basically 
> breaking any e.g. cron job based solution kicking off -pr based repairs on 
> several nodes concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13663) Cassandra 3.10 crashes without dump

[Ricardo Bartolome (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171766#comment-16171766
 ] 

Ricardo Bartolome edited comment on CASSANDRA-13663 at 9/19/17 2:08 PM:


Does anybody have news about this issue?

We are experiencing a similar issue even in our case we don't see any 
oom-killer errors. Our scenario is:
* 12 x i3.2xlarge instances (8 CPU, 64GB memory)
* Storage per node is ~400GB
* Cassandra 3.9 (looking for an upgrade to 3.10, but nothing appears listed in 
the CHANGELOG related to this issue, and now we found this issue)
* Oracle JVM build 1.8.0_112-b15

We also have 1-2 dead nodes a week. We have been enabling HeapDumps on several 
nodes to help to identify but so far didn't reproduce in the nodes that have it 
enabled (neither if it will contain some useful information, if the problem is 
off-heap).

Some off-heap memory usage statistics gathered through JMX exploring the 
following beans:
* org.apache.cassandra.metrics:name=BloomFilterOffHeapMemoryUsed,type=Table
* org.apache.cassandra.metrics:name=AllMemtablesOffHeapSize,type=Table
* 
org.apache.cassandra.metrics:name=CompressionMetadataOffHeapMemoryUsed,type=Table

h4. BloomFilterOffHeapMemoryUsed (~ 1.6GB)
{code}
Value = 1619193928;
Value = 1546767024;
Value = 1669879216;
Value = 1576772336;
Value = 1567804792;
Value = 1605097824;
Value = 1608551904;
Value = 1502500424;
Value = 1363705192;
Value = 1259389280;
Value = 1671282736;
{code}

h4. AllMemtablesOffHeapSize (~700MB)
{code}
Value = 692597111;
Value = 617691154;
Value = 693412363;
Value = 708732630;
Value = 664297343;
Value = 705367430;
Value = 626936323;
Value = 652724309;
Value = 700223457;
Value = 666516571;
Value = 682268720;
{code}

h4. CompressionMetadataOffHeapMemoryUsed (~110MB)
{code}
Value = 111307336;
Value = 105718312;
Value = 110638576;
Value = 111370032;
Value = 105979296;
Value = 108963456;
Value = 22216;
Value = 99788200;
Value = 95279232;
Value = 106130392;
Value = 113217400;
{code}

Any idea what else I can look at?



was (Author: ricbartm):
Does anybody have news about this issue?

We are experiencing a similar issue even in our case we don't see any 
oom-killer errors. Our scenario is:
* 12 x i3.2xlarge instances (8 CPU, 64GB memory)
* Storage per node is ~400GB
* Cassandra 3.9 (looking for an upgrade, but nothing appears listed in the 
CHANGELOG related to this issue)
* Oracle JVM build 1.8.0_112-b15

We also have 1-2 dead nodes a week. We have been enabling HeapDumps on several 
nodes to help to identify but so far didn't reproduce in the nodes that have it 
enabled (neither if it will contain some useful information, if the problem is 
off-heap).

Some off-heap memory usage statistics gathered through JMX exploring the 
following beans:
* org.apache.cassandra.metrics:name=BloomFilterOffHeapMemoryUsed,type=Table
* org.apache.cassandra.metrics:name=AllMemtablesOffHeapSize,type=Table
* 
org.apache.cassandra.metrics:name=CompressionMetadataOffHeapMemoryUsed,type=Table

h4. BloomFilterOffHeapMemoryUsed (~ 1.6GB)
{code}
Value = 1619193928;
Value = 1546767024;
Value = 1669879216;
Value = 1576772336;
Value = 1567804792;
Value = 1605097824;
Value = 1608551904;
Value = 1502500424;
Value = 1363705192;
Value = 1259389280;
Value = 1671282736;
{code}

h4. AllMemtablesOffHeapSize (~700MB)
{code}
Value = 692597111;
Value = 617691154;
Value = 693412363;
Value = 708732630;
Value = 664297343;
Value = 705367430;
Value = 626936323;
Value = 652724309;
Value = 700223457;
Value = 666516571;
Value = 682268720;
{code}

h4. CompressionMetadataOffHeapMemoryUsed (~110MB)
{code}
Value = 111307336;
Value = 105718312;
Value = 110638576;
Value = 111370032;
Value = 105979296;
Value = 108963456;
Value = 22216;
Value = 99788200;
Value = 95279232;
Value = 106130392;
Value = 113217400;
{code}

Any idea what else I can look at?


> Cassandra 3.10 crashes without dump
> ---
>
> Key: CASSANDRA-13663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13663
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Matthias Otto
>Priority: Minor
> Attachments: 2017-07-04 10_48_34-CloudWatch Management Console.png, 
> cassandra debug.log, cassandra system.log, RamUsageExamle1.png, 
> RamUsageExample2.png
>
>
> Hello. My company runs a 5 node Cassandra cluster. For the last few weeks, we 
> have had a sporadic issue where one of the servers crashes without creating a 
> dump file and without any error messages in the logs. If one restarts the 
> service (which we have by now scripted to happen automatically), the servers 
> resumes work with no complaint.
> Log files of the time of the last crash are attached, thou again they do not 
> log any crash happening.
> Regarding out setup, we are running these servers on AMazon AWS, with 3 
> volumes 

[jira] [Commented] (CASSANDRA-13663) Cassandra 3.10 crashes without dump

[Ricardo Bartolome (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171766#comment-16171766
 ] 

Ricardo Bartolome commented on CASSANDRA-13663:
---

Does anybody have news about this issue?

We are experiencing a similar issue even in our case we don't see any 
oom-killer errors. Our scenario is:
* 12 x i3.2xlarge instances (8 CPU, 64GB memory)
* Storage per node is ~400GB
* Cassandra 3.9 (looking for an upgrade, but nothing appears listed in the 
CHANGELOG related to this issue)
* Oracle JVM build 1.8.0_112-b15

We also have 1-2 dead nodes a week. We have been enabling HeapDumps on several 
nodes to help to identify but so far didn't reproduce in the nodes that have it 
enabled (neither if it will contain some useful information, if the problem is 
off-heap).

Some off-heap memory usage statistics gathered through JMX exploring the 
following beans:
* org.apache.cassandra.metrics:name=BloomFilterOffHeapMemoryUsed,type=Table
* org.apache.cassandra.metrics:name=AllMemtablesOffHeapSize,type=Table
* 
org.apache.cassandra.metrics:name=CompressionMetadataOffHeapMemoryUsed,type=Table

h4. BloomFilterOffHeapMemoryUsed (~ 1.6GB)
{code}
Value = 1619193928;
Value = 1546767024;
Value = 1669879216;
Value = 1576772336;
Value = 1567804792;
Value = 1605097824;
Value = 1608551904;
Value = 1502500424;
Value = 1363705192;
Value = 1259389280;
Value = 1671282736;
{code}

h4. AllMemtablesOffHeapSize (~700MB)
{code}
Value = 692597111;
Value = 617691154;
Value = 693412363;
Value = 708732630;
Value = 664297343;
Value = 705367430;
Value = 626936323;
Value = 652724309;
Value = 700223457;
Value = 666516571;
Value = 682268720;
{code}

h4. CompressionMetadataOffHeapMemoryUsed (~110MB)
{code}
Value = 111307336;
Value = 105718312;
Value = 110638576;
Value = 111370032;
Value = 105979296;
Value = 108963456;
Value = 22216;
Value = 99788200;
Value = 95279232;
Value = 106130392;
Value = 113217400;
{code}

Any idea what else I can look at?


> Cassandra 3.10 crashes without dump
> ---
>
> Key: CASSANDRA-13663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13663
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Matthias Otto
>Priority: Minor
> Attachments: 2017-07-04 10_48_34-CloudWatch Management Console.png, 
> cassandra debug.log, cassandra system.log, RamUsageExamle1.png, 
> RamUsageExample2.png
>
>
> Hello. My company runs a 5 node Cassandra cluster. For the last few weeks, we 
> have had a sporadic issue where one of the servers crashes without creating a 
> dump file and without any error messages in the logs. If one restarts the 
> service (which we have by now scripted to happen automatically), the servers 
> resumes work with no complaint.
> Log files of the time of the last crash are attached, thou again they do not 
> log any crash happening.
> Regarding out setup, we are running these servers on AMazon AWS, with 3 
> volumes per server, one for the system, one for data and one for the 
> commitlog. When a crash happens, we can observe a sudden spike of read 
> activity on the commitlog volume. All of these have ample free space. 
> Aspecially the system volume has more then enough free space so that a dump 
> could be written.
> The servers are Ubuntu 16.04 servers and Cassandra is installed from the 
> apt-get packet for version 3.10.
> It is worth noting that these crashes happen more often when nodetool is 
> running either repair job or a backup job, but this is by no means always the 
> case. As for frequency, we have had about 1-2 crashes per week for the last 
> month.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13885) Allow to run full repairs in 3.0 without additional cost of anti-compaction

[Thomas Steinmaurer (JIRA)]

Thomas Steinmaurer created CASSANDRA-13885:
--

 Summary: Allow to run full repairs in 3.0 without additional cost 
of anti-compaction
 Key: CASSANDRA-13885
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13885
 Project: Cassandra
  Issue Type: Bug
Reporter: Thomas Steinmaurer


This ticket is basically the result of the discussion in Cassandra user list: 
https://www.mail-archive.com/user@cassandra.apache.org/msg53562.html

I was asked to open a ticket by Paulo Motta to think about back-porting running 
full repairs without the additional cost of anti-compaction.

Basically there is no way in 3.0 to run full repairs from several nodes 
concurrently without troubles caused by (overlapping?) anti-compactions. Coming 
from 2.1 this is a major change from an operational POV, basically breaking any 
e.g. cron job based solution kicking off -pr based repairs on several nodes 
concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13404) Hostname verification for client-to-node encryption

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171617#comment-16171617
 ] 

Jason Brown commented on CASSANDRA-13404:
-

[~eperott] and [~Jan Karlsson] I was planning on coming back to this after 
CASSANDRA-8457, and now that it's out of the way we can pick it up. At a 
minimum we'll need to rebase the patch (should be trivial), and we should 
probably add some dtests. New dtests should probably go in 
{{native_transport_ssl_test.py}}, and there's examples in 
{{sslnodetonode_test.py}} for doing TLS-type stuff like endpoint verification.



> Hostname verification for client-to-node encryption
> ---
>
> Key: CASSANDRA-13404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13404-trunk.txt
>
>
> Similarily to CASSANDRA-9220, Cassandra should support hostname verification 
> for client-node connections.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13404) Hostname verification for client-to-node encryption

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171617#comment-16171617
 ] 

Jason Brown edited comment on CASSANDRA-13404 at 9/19/17 12:40 PM:
---

[~eperott] and [~Jan Karlsson] I was planning on coming back to this after 
CASSANDRA-8457, and now that it's out of the way we can pick it up. At a 
minimum we'll need to rebase the patch (should be trivial), and we should 
probably add some dtests. New dtests should probably go in 
{{native_transport_ssl_test.py}}, and there's examples in 
{{sslnodetonode_test.py}} for doing TLS-type stuff like endpoint verification. 
Meanwhile I"ll page all of this back in...




was (Author: jasobrown):
[~eperott] and [~Jan Karlsson] I was planning on coming back to this after 
CASSANDRA-8457, and now that it's out of the way we can pick it up. At a 
minimum we'll need to rebase the patch (should be trivial), and we should 
probably add some dtests. New dtests should probably go in 
{{native_transport_ssl_test.py}}, and there's examples in 
{{sslnodetonode_test.py}} for doing TLS-type stuff like endpoint verification.



> Hostname verification for client-to-node encryption
> ---
>
> Key: CASSANDRA-13404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13404-trunk.txt
>
>
> Similarily to CASSANDRA-9220, Cassandra should support hostname verification 
> for client-node connections.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13149) AssertionError prepending to a list

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-13149:

Status: In Progress  (was: Patch Available)

Mostly the changes look good. I do think though that we need to handle the case 
where the list of elements to prepend > {{MAX_NANOS}} better than just throwing 
an assertion error (obviously not a good idea to do it, but it is allowed). 

This is probably more contentious, but there's also some potential for misuse 
here as nothing currently stops us overflowing the range of the PT instance 
except the construction of the for loop. We could make PT.nanos private and add 
a {{getNanosPlusDelta(int)}} instance method that  asserts that we don't try to 
use more that were originally allocated.

Nits: 
* typo Lists line #290 {{miilisToUse}}
* typo ListsTest line #65 {{_Mulitple}}
* in other fixtures where we've used the snake case name format, the names tend 
to have the {{test}} prefix 
({{BufferPoolTest/BTreeTest/SegmentedFileTest/etc}}, only 
{{RepariMessageSerializationsTest}} deviates from that pattern currently).


> AssertionError prepending to a list
> ---
>
> Key: CASSANDRA-13149
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13149
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: 3.0.8
>Reporter: Steven Warren
>Assignee: Jason Brown
>
> Prepending to a list produces the following AssertionError randomly. Changing 
> the update to append (and sort in the client) works around the issue.
> {code}
> java.lang.AssertionError: null
>   at 
> org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275) 
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430) 
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_101]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [apache-cassandra-3.0.8.jar:3.0.8]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12245) initial view build can be parallel

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-12245:

Status: Open  (was: Patch Available)

> initial view build can be parallel
> --
>
> Key: CASSANDRA-12245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12245
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Materialized Views
>Reporter: Tom van der Woerdt
>Assignee: Andrés de la Peña
> Fix For: 4.x
>
>
> On a node with lots of data (~3TB) building a materialized view takes several 
> weeks, which is not ideal. It's doing this in a single thread.
> There are several potential ways this can be optimized :
>  * do vnodes in parallel, instead of going through the entire range in one 
> thread
>  * just iterate through sstables, not worrying about duplicates, and include 
> the timestamp of the original write in the MV mutation. since this doesn't 
> exclude duplicates it does increase the amount of work and could temporarily 
> surface ghost rows (yikes) but I guess that's why they call it eventual 
> consistency. doing it this way can avoid holding references to all tables on 
> disk, allows parallelization, and removes the need to check other sstables 
> for existing data. this is essentially the 'do a full repair' path



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12245) initial view build can be parallel

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171536#comment-16171536
 ] 

Paulo Motta commented on CASSANDRA-12245:
-

Finally getting to this after a while, sorry for the delay. Thanks for the 
update! Had another look at the patch and this is looking much better, see some 
follow-up comments below:

bq. I have moved the methods to split the ranges to the Splitter, reusing its 
valueForToken method. Tests here.

Awesome, looks much better now! It seems like the way the number of tokens in a 
range was computed by {{abs(range.right - range.left)}} may not work correctly 
for some wrap-around cases, as shown by [this test 
case|https://github.com/pauloricardomg/cassandra/blob/2760bbbc25a2ad9a9bbf9d29a0dc19e1e3bfb237/test/unit/org/apache/cassandra/dht/SplitterTest.java#L184].
 Even though this shouldn't break when local ranges are used , I fixed it on 
[this 
commit|https://github.com/pauloricardomg/cassandra/commit/2760bbbc25a2ad9a9bbf9d29a0dc19e1e3bfb237]
 to make sure split works correctly for wrap-around ranges. Can you confirm 
this is correct?

Other than that, it seems like you added unit tests only for 
{{Murmur3Partitioner}}, would you mind extending {{testSplit()}} to 
{{RandomPartitioner}}?

bq. Agree. I have added a new dedicated executor in the CompactionManager, 
similar to the executors used for validation and cache cleanup. The concurrency 
of this executor is determined by the new config property 
concurrent_materialized_view_builders, which defaults to a perhaps too much 
conservative value of 1. This property can be modified through both JMX and the 
new setconcurrentviewbuilders and getconcurrentviewbuilders nodetool commands. 
These commands are tested here.

I think having a dedicated executor will ensure view building doesn't compete 
with compactions for the compaction executor, good job! One problem I see 
though is that if the user is finding its view building slow it will try to 
increase the number of concurrent view builders via nodetool, but it will have 
no effect since the range was split in the previously number of concurrent view 
builders. Given this will be a pretty common scenario for large datasets, how 
about splitting the range in multiple smaller tasks, so that if the user 
increases {{concurrent_view_builders}} the other tasks immediately start 
executing?

We could use a simple approach of splitting the local range in let's say 1000 
hard-coded parts, or be smarter and make each split have ~100MB or so. In this 
way we can keep {{concurrent_materialized_view_builders=1}} by default, and 
users with large base tables are able to increase it and see immediate effect 
via nodetool. WDYT?

bq. I would prefer to do this in another ticket.

Agreed.

bq. I have moved the marking of system tables (and the retries in case of 
failure) from the ViewBuilderTask to the ViewBuilder, using a callback to do 
the marking. I think the code is clearer this way.

Great, looks much cleaner indeed! One minor thing is that if there's a failure 
after some {{ViewBuildTasks}} were completed, it will resume that subtask from 
its last token while it already finished. Could we maybe set the last_token = 
end_token when the task is finished to flag it was already finished and avoid 
resuming the task when that is the case?

bq. Updated here. It also uses a byteman script to make sure that the MV build 
isn't finished before stopping the cluster, which is more likely to happen.

The dtest looks mostly good, except for the following nits:
* {{concurrent_materialized_view_builders=1}} when the nodes are restarted. Can 
you set the configuration value during cluster setup phase (instead of setting 
via nodetool) to make sure the restarted view builds will be parallel?
* can probably use {{self._wait_for_view("ks", "t_by_v")}} 
[here|https://github.com/adelapena/cassandra-dtest/blob/cf893982c361fd7b6018b2570d3a5a33badd5424/materialized_views_test.py#L982]
* We cannot ensure key 1 was not built 
[here|https://github.com/adelapena/cassandra-dtest/blob/cf893982c361fd7b6018b2570d3a5a33badd5424/materialized_views_test.py#L980]
 which may cause flakiness, so it's probably better to check for 
{{self.assertNotEqual(len(list(session.execute("SELECT count\(*\) FROM 
t_by_v;"))), 1)}} or something like that.
* It would be nice to check that the view build was actually removed on 
restart, by checking for the log entry {{Resuming view build for range}}

bq. I'm not sure about if it still makes sense for the builder task to extend 
CompactionInfo.Holder. If so, I'm neither sure about how to use 
prevToken.size(range.right) (that returns a double) to create CompationInfo 
objects. WDYT?

I think it's still useful to show view build progress via {{nodetool 
compactionstats}}, so I created a new method {{Splitter.positionInRange(Token 
token, Range range)}} which gives the 

[jira] [Updated] (CASSANDRA-13149) AssertionError prepending to a list

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-13149:

Reviewer: Sam Tunnicliffe

> AssertionError prepending to a list
> ---
>
> Key: CASSANDRA-13149
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13149
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: 3.0.8
>Reporter: Steven Warren
>Assignee: Jason Brown
>
> Prepending to a list produces the following AssertionError randomly. Changing 
> the update to append (and sort in the client) works around the issue.
> {code}
> java.lang.AssertionError: null
>   at 
> org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275) 
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430) 
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
>  ~[apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_101]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  [apache-cassandra-3.0.8.jar:3.0.8]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [apache-cassandra-3.0.8.jar:3.0.8]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-10857) Allow dropping COMPACT STORAGE flag from tables in 3.X

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-10857:

Fix Version/s: (was: 4.0)
   3.0.x

> Allow dropping COMPACT STORAGE flag from tables in 3.X
> --
>
> Key: CASSANDRA-10857
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10857
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL, Distributed Metadata
>Reporter: Aleksey Yeschenko
>Assignee: Alex Petrov
>Priority: Blocker
> Fix For: 3.0.x, 3.11.x
>
>
> Thrift allows users to define flexible mixed column families - where certain 
> columns would have explicitly pre-defined names, potentially non-default 
> validation types, and be indexed.
> Example:
> {code}
> create column family foo
> and default_validation_class = UTF8Type
> and column_metadata = [
> {column_name: bar, validation_class: Int32Type, index_type: KEYS},
> {column_name: baz, validation_class: UUIDType, index_type: KEYS}
> ];
> {code}
> Columns named {{bar}} and {{baz}} will be validated as {{Int32Type}} and 
> {{UUIDType}}, respectively, and be indexed. Columns with any other name will 
> be validated by {{UTF8Type}} and will not be indexed.
> With CASSANDRA-8099, {{bar}} and {{baz}} would be mapped to static columns 
> internally. However, being {{WITH COMPACT STORAGE}}, the table will only 
> expose {{bar}} and {{baz}} columns. Accessing any dynamic columns (any column 
> not named {{bar}} and {{baz}}) right now requires going through Thrift.
> This is blocking Thrift -> CQL migration for users who have mixed 
> dynamic/static column families. That said, it *shouldn't* be hard to allow 
> users to drop the {{compact}} flag to expose the table as it is internally 
> now, and be able to access all columns.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13880) Fix short read protection for tables with no clustering columns

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171476#comment-16171476
 ] 

Aleksey Yeschenko commented on CASSANDRA-13880:
---

Thanks.

CircleCI run for [3.0|https://circleci.com/gh/iamaleksey/cassandra/34] has a 
bunch of annoying MV timeout issues again, and 
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/326/testReport/]
 have those annoying git clone issues, again, but everything seems alright 
otherwise.

Committed to 3.0 as 
[35e32f20ba8cba6cbfb1bee4252c0edd8684cdb1|https://github.com/apache/cassandra/commit/35e32f20ba8cba6cbfb1bee4252c0edd8684cdb1]
 and merged with 3.11 and trunk. Dtest committed as 
[163f82c2db0e86d4dd8f312b291ccd094891b986|https://github.com/apache/cassandra-dtest/commit/163f82c2db0e86d4dd8f312b291ccd094891b986].

> Fix short read protection for tables with no clustering columns
> ---
>
> Key: CASSANDRA-13880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13880
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.15, 3.11.1
>
>
> CASSANDRA-12872 fixed counting replica rows, so that we do now fetch more 
> than one extra row if necessary.
> Fixing the issue caused consistency_test.py:TestConsistency.test_13747 to 
> start failing, by exposing a bug in the way we handle empty clusterings.
> When {{moreContents()}} asks for another row and {{lastClustering}} is 
> {{EMPTY}}, the response again (and again) contains the row with {{EMPTY}} 
> clustering.
> SRP assumes it’s a new row, counts it as one, gets confused and keeps asking 
> for more, in a loop, again and again.
> Arguably, a response to a read command with the following non-inclusive 
> {{ClusteringIndexFilter}}:
> {code}
> command.clusteringIndexFilter(partitionKey).forPaging(metadata.comparator, 
> Clustering.EMPTY, false);
> {code}
> ... should return nothing at all rather than a row with an empty clustering.
> Also arguably, SRP should not even attempt to fetch more rows if 
> {{lastClustering == Clustering.EMPTY}}. In a partition key only column
> we shouldn’t expect any more rows.
> This JIRA is to fix the latter issue on SRP side - to modify SRP logic to 
> short-circuit execution if {{lastClustering}} was an {{EMPTY}} one instead of 
> querying pointlessly for non-existent extra rows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13880) Fix short read protection for tables with no clustering columns

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13880:
--
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 3.0.x)
   3.11.1
   3.0.15
   Status: Resolved  (was: Patch Available)

> Fix short read protection for tables with no clustering columns
> ---
>
> Key: CASSANDRA-13880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13880
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.15, 3.11.1
>
>
> CASSANDRA-12872 fixed counting replica rows, so that we do now fetch more 
> than one extra row if necessary.
> Fixing the issue caused consistency_test.py:TestConsistency.test_13747 to 
> start failing, by exposing a bug in the way we handle empty clusterings.
> When {{moreContents()}} asks for another row and {{lastClustering}} is 
> {{EMPTY}}, the response again (and again) contains the row with {{EMPTY}} 
> clustering.
> SRP assumes it’s a new row, counts it as one, gets confused and keeps asking 
> for more, in a loop, again and again.
> Arguably, a response to a read command with the following non-inclusive 
> {{ClusteringIndexFilter}}:
> {code}
> command.clusteringIndexFilter(partitionKey).forPaging(metadata.comparator, 
> Clustering.EMPTY, false);
> {code}
> ... should return nothing at all rather than a row with an empty clustering.
> Also arguably, SRP should not even attempt to fetch more rows if 
> {{lastClustering == Clustering.EMPTY}}. In a partition key only column
> we shouldn’t expect any more rows.
> This JIRA is to fix the latter issue on SRP side - to modify SRP logic to 
> short-circuit execution if {{lastClustering}} was an {{EMPTY}} one instead of 
> querying pointlessly for non-existent extra rows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra-dtest git commit: Fix short read protection for tables with no clustering columns

[aleksey]

Repository: cassandra-dtest
Updated Branches:
  refs/heads/master 6220394e8 -> 163f82c2d


Fix short read protection for tables with no clustering columns

patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-13880


Project: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/commit/163f82c2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/tree/163f82c2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/diff/163f82c2

Branch: refs/heads/master
Commit: 163f82c2db0e86d4dd8f312b291ccd094891b986
Parents: 6220394
Author: Aleksey Yeschenko 
Authored: Mon Sep 18 16:14:24 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Mon Sep 18 17:33:11 2017 +0100

--
 consistency_test.py | 35 +++
 1 file changed, 35 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/163f82c2/consistency_test.py
--
diff --git a/consistency_test.py b/consistency_test.py
index 27e5d01..407873c 100644
--- a/consistency_test.py
+++ b/consistency_test.py
@@ -774,6 +774,41 @@ class TestAccuracy(TestHelper):
 class TestConsistency(Tester):
 
 @since('3.0')
+def test_13880(self):
+"""
+@jira_ticket CASSANDRA-13880
+"""
+cluster = self.cluster
+
+# disable hinted handoff and set batch commit log so this doesn't 
interfere with the test
+cluster.set_configuration_options(values={'hinted_handoff_enabled': 
False})
+cluster.set_batch_commitlog(enabled=True)
+
+cluster.populate(2).start(wait_other_notice=True)
+node1, node2 = cluster.nodelist()
+
+session = self.patient_cql_connection(node1)
+
+query = "CREATE KEYSPACE IF NOT EXISTS test WITH replication = 
{'class': 'NetworkTopologyStrategy', 'datacenter1': 2};"
+session.execute(query)
+
+query = "CREATE TABLE IF NOT EXISTS test.test (id int PRIMARY KEY);"
+session.execute(query)
+
+stmt = SimpleStatement("INSERT INTO test.test (id) VALUES (0);",
+   consistency_level = ConsistencyLevel.ALL)
+session.execute(stmt)
+
+# with node2 down and hints disabled, delete the partition on node1
+node2.stop(wait_other_notice=True)
+session.execute("DELETE FROM test.test WHERE id = 0;");
+node2.start(wait_other_notice=True)
+
+# with both nodes up, do a CL.ALL query with per partition limit of 1;
+# prior to CASSANDRA-13880 this would cause short read protection to 
loop forever
+assert_none(session, "SELECT DISTINCT id FROM test.test WHERE id = 
0;", cl=ConsistencyLevel.ALL)
+
+@since('3.0')
 def test_13747(self):
 """
 @jira_ticket CASSANDRA-13747


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Fix short read protection for tables with no clustering columns

[aleksey]

Fix short read protection for tables with no clustering columns

patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-13880


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35e32f20
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35e32f20
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35e32f20

Branch: refs/heads/trunk
Commit: 35e32f20ba8cba6cbfb1bee4252c0edd8684cdb1
Parents: bb3b332
Author: Aleksey Yeschenko 
Authored: Mon Sep 18 14:01:19 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Mon Sep 18 16:13:43 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 84ef845..74e70e1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)
  * Fix counter application order in short read protection (CASSANDRA-12872)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 580fd8b..99399a3 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -510,6 +510,8 @@ public class DataResolver extends ResponseResolver
 @Override
 public UnfilteredRowIterator moreContents()
 {
+assert !postReconciliationCounter.isDoneForPartition();
+
 // We have a short read if the node this is the result of has 
returned the requested number of
 // rows for that partition (i.e. it has stopped returning 
results due to the limit), but some of
 // those results haven't made it in the final result 
post-reconciliation due to other nodes
@@ -522,9 +524,13 @@ public class DataResolver extends ResponseResolver
 // skipped during reconciliation.
 if (lastCount == counter.counted() || 
!counter.isDoneForPartition())
 return null;
-lastCount = counter.counted();
 
-assert !postReconciliationCounter.isDoneForPartition();
+// clustering of the last row returned is empty, meaning that 
there is only one row per partition,
+// and we already have it.
+if (lastClustering == Clustering.EMPTY)
+return null;
+
+lastCount = counter.counted();
 
 // We need to try to query enough additional results to 
fulfill our query, but because we could still
 // get short reads on that additional query, just querying the 
number of results we miss may not be


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[aleksey]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb766921
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb766921
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb766921

Branch: refs/heads/trunk
Commit: eb7669215938f29cfb8230fba0b382742b93bd48
Parents: bcd01f2 594f1c1
Author: Aleksey Yeschenko 
Authored: Tue Sep 19 11:18:28 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Tue Sep 19 11:18:28 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb766921/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb766921/src/java/org/apache/cassandra/service/DataResolver.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594f1c1d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594f1c1d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594f1c1d

Branch: refs/heads/cassandra-3.11
Commit: 594f1c1dfba44530e2b25f83a1e9d71f79aba7f6
Parents: 2556676 35e32f2
Author: Aleksey Yeschenko 
Authored: Tue Sep 19 11:18:20 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Tue Sep 19 11:18:20 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/594f1c1d/CHANGES.txt
--
diff --cc CHANGES.txt
index 54117df,74e70e1..01955f6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,5 +1,16 @@@
 -3.0.15
 +3.11.1
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
   * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
   * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)
   * Fix counter application order in short read protection (CASSANDRA-12872)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/594f1c1d/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --cc src/java/org/apache/cassandra/service/DataResolver.java
index 5708832,99399a3..4b0bd3c
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@@ -536,11 -522,15 +538,15 @@@ public class DataResolver extends Respo
  // Also note that we only get here once all the results for 
this node have been returned, and so
  // if the node had returned the requested number but we still 
get there, it imply some results were
  // skipped during reconciliation.
 -if (lastCount == counter.counted() || 
!counter.isDoneForPartition())
 +if (lastCount == counted(counter) || 
!counter.isDoneForPartition())
  return null;
- lastCount = counted(counter);
  
- assert !postReconciliationCounter.isDoneForPartition();
+ // clustering of the last row returned is empty, meaning that 
there is only one row per partition,
+ // and we already have it.
+ if (lastClustering == Clustering.EMPTY)
+ return null;
+ 
 -lastCount = counter.counted();
++lastCount = counted(counter);
  
  // We need to try to query enough additional results to 
fulfill our query, but because we could still
  // get short reads on that additional query, just querying 
the number of results we miss may not be


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Fix short read protection for tables with no clustering columns

[aleksey]

Fix short read protection for tables with no clustering columns

patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-13880


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35e32f20
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35e32f20
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35e32f20

Branch: refs/heads/cassandra-3.11
Commit: 35e32f20ba8cba6cbfb1bee4252c0edd8684cdb1
Parents: bb3b332
Author: Aleksey Yeschenko 
Authored: Mon Sep 18 14:01:19 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Mon Sep 18 16:13:43 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 84ef845..74e70e1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)
  * Fix counter application order in short read protection (CASSANDRA-12872)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 580fd8b..99399a3 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -510,6 +510,8 @@ public class DataResolver extends ResponseResolver
 @Override
 public UnfilteredRowIterator moreContents()
 {
+assert !postReconciliationCounter.isDoneForPartition();
+
 // We have a short read if the node this is the result of has 
returned the requested number of
 // rows for that partition (i.e. it has stopped returning 
results due to the limit), but some of
 // those results haven't made it in the final result 
post-reconciliation due to other nodes
@@ -522,9 +524,13 @@ public class DataResolver extends ResponseResolver
 // skipped during reconciliation.
 if (lastCount == counter.counted() || 
!counter.isDoneForPartition())
 return null;
-lastCount = counter.counted();
 
-assert !postReconciliationCounter.isDoneForPartition();
+// clustering of the last row returned is empty, meaning that 
there is only one row per partition,
+// and we already have it.
+if (lastClustering == Clustering.EMPTY)
+return null;
+
+lastCount = counter.counted();
 
 // We need to try to query enough additional results to 
fulfill our query, but because we could still
 // get short reads on that additional query, just querying the 
number of results we miss may not be


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix short read protection for tables with no clustering columns

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 bb3b332c4 -> 35e32f20b
  refs/heads/cassandra-3.11 255667687 -> 594f1c1df
  refs/heads/trunk bcd01f23d -> eb7669215


Fix short read protection for tables with no clustering columns

patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-13880


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35e32f20
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35e32f20
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35e32f20

Branch: refs/heads/cassandra-3.0
Commit: 35e32f20ba8cba6cbfb1bee4252c0edd8684cdb1
Parents: bb3b332
Author: Aleksey Yeschenko 
Authored: Mon Sep 18 14:01:19 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Mon Sep 18 16:13:43 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 84ef845..74e70e1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)
  * Fix counter application order in short read protection (CASSANDRA-12872)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/35e32f20/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 580fd8b..99399a3 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -510,6 +510,8 @@ public class DataResolver extends ResponseResolver
 @Override
 public UnfilteredRowIterator moreContents()
 {
+assert !postReconciliationCounter.isDoneForPartition();
+
 // We have a short read if the node this is the result of has 
returned the requested number of
 // rows for that partition (i.e. it has stopped returning 
results due to the limit), but some of
 // those results haven't made it in the final result 
post-reconciliation due to other nodes
@@ -522,9 +524,13 @@ public class DataResolver extends ResponseResolver
 // skipped during reconciliation.
 if (lastCount == counter.counted() || 
!counter.isDoneForPartition())
 return null;
-lastCount = counter.counted();
 
-assert !postReconciliationCounter.isDoneForPartition();
+// clustering of the last row returned is empty, meaning that 
there is only one row per partition,
+// and we already have it.
+if (lastClustering == Clustering.EMPTY)
+return null;
+
+lastCount = counter.counted();
 
 // We need to try to query enough additional results to 
fulfill our query, but because we could still
 // get short reads on that additional query, just querying the 
number of results we miss may not be


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594f1c1d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594f1c1d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594f1c1d

Branch: refs/heads/trunk
Commit: 594f1c1dfba44530e2b25f83a1e9d71f79aba7f6
Parents: 2556676 35e32f2
Author: Aleksey Yeschenko 
Authored: Tue Sep 19 11:18:20 2017 +0100
Committer: Aleksey Yeschenko 
Committed: Tue Sep 19 11:18:20 2017 +0100

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/service/DataResolver.java | 10 --
 2 files changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/594f1c1d/CHANGES.txt
--
diff --cc CHANGES.txt
index 54117df,74e70e1..01955f6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,5 +1,16 @@@
 -3.0.15
 +3.11.1
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
   * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
   * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)
   * Fix counter application order in short read protection (CASSANDRA-12872)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/594f1c1d/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --cc src/java/org/apache/cassandra/service/DataResolver.java
index 5708832,99399a3..4b0bd3c
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@@ -536,11 -522,15 +538,15 @@@ public class DataResolver extends Respo
  // Also note that we only get here once all the results for 
this node have been returned, and so
  // if the node had returned the requested number but we still 
get there, it imply some results were
  // skipped during reconciliation.
 -if (lastCount == counter.counted() || 
!counter.isDoneForPartition())
 +if (lastCount == counted(counter) || 
!counter.isDoneForPartition())
  return null;
- lastCount = counted(counter);
  
- assert !postReconciliationCounter.isDoneForPartition();
+ // clustering of the last row returned is empty, meaning that 
there is only one row per partition,
+ // and we already have it.
+ if (lastClustering == Clustering.EMPTY)
+ return null;
+ 
 -lastCount = counter.counted();
++lastCount = counted(counter);
  
  // We need to try to query enough additional results to 
fulfill our query, but because we could still
  // get short reads on that additional query, just querying 
the number of results we miss may not be


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13404) Hostname verification for client-to-node encryption

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171305#comment-16171305
 ] 

Per Otterström commented on CASSANDRA-13404:


Let me (colleague of Jan) add a little context to describe why we want hostname 
verification on the server side. In our deployments all peers (both clients and 
servers) will automatically get unique certificates signed for them. This allow 
us to limit access such that only trusted clients can connect to Cassandra. 
Hostname verification on the server side would improve security in that no one 
is able to copy a certificate from a client node to another location, and then 
use it to manipulate data in Cassandra.

The patch provided by Jan contain two different changes:

- The first is about setting the endpoint identification algorithm. This seem 
pretty straight forward as it is optional and in line with what we already have 
on the server-to-server communication.

- The second is about passing the remote peer information into the SSLEngine as 
it is created. This is necessary to make remote peer information available to 
the TrustManager when it is time to validate the certificate chain. And so 
without this second modification, the first change doesn't make much sense. I 
understand this change is in conflict with a potential performance improvement 
by reusing SSLContext's, but as Jan is pointing out, this patch set is only 
associating peer specific data with the SSLEngine's created from the 
SSLContext. The SSLContext itself could still be reused.

I understand this functionality is valuable to a very small group of users, but 
as it is optional I can not see the harm of adding it. Thoughts?

> Hostname verification for client-to-node encryption
> ---
>
> Key: CASSANDRA-13404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13404-trunk.txt
>
>
> Similarily to CASSANDRA-9220, Cassandra should support hostname verification 
> for client-node connections.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-10404) Node to Node encryption transitional mode

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-10404:
---
Since Version:   (was: 4.0)
 Reviewer: Stefan Podkowinski
Fix Version/s: 4.x

> Node to Node encryption transitional mode
> -
>
> Key: CASSANDRA-10404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Tom Lewis
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Create a transitional mode for encryption that allows encrypted and 
> unencrypted traffic node-to-node during a change over to encryption from 
> unencrypted. This alleviates downtime during the switch.
>  This is similar to CASSANDRA-10559 which is intended for client-to-node



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13651) Large amount of CPU used by epoll_wait(.., .., .., 0)

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171291#comment-16171291
 ] 

Corentin Chary commented on CASSANDRA-13651:


Thanks for double-checking that, I pushed the wrong version. This should now be 
fixed.

> Large amount of CPU used by epoll_wait(.., .., .., 0)
> -
>
> Key: CASSANDRA-13651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13651
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 4.x
>
> Attachments: cpu-usage.png
>
>
> I was trying to profile Cassandra under my workload and I kept seeing this 
> backtrace:
> {code}
> epollEventLoopGroup-2-3 State: RUNNABLE CPU usage on sample: 240ms
> io.netty.channel.epoll.Native.epollWait0(int, long, int, int) Native.java 
> (native)
> io.netty.channel.epoll.Native.epollWait(int, EpollEventArray, int) 
> Native.java:111
> io.netty.channel.epoll.EpollEventLoop.epollWait(boolean) 
> EpollEventLoop.java:230
> io.netty.channel.epoll.EpollEventLoop.run() EpollEventLoop.java:254
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run() 
> SingleThreadEventExecutor.java:858
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run() 
> DefaultThreadFactory.java:138
> java.lang.Thread.run() Thread.java:745
> {code}
> At fist I though that the profiler might not be able to profile native code 
> properly, but I wen't further and I realized that most of the CPU was used by 
> {{epoll_wait()}} calls with a timeout of zero.
> Here is the output of perf on this system, which confirms that most of the 
> overhead was with timeout == 0.
> {code}
> Samples: 11M of event 'syscalls:sys_enter_epoll_wait', Event count (approx.): 
> 11594448
> Overhead  Trace output
>   
>  ◆
>   90.06%  epfd: 0x0047, events: 0x7f5588c0c000, maxevents: 0x2000, 
> timeout: 0x   
> ▒
>5.77%  epfd: 0x00b5, events: 0x7fca419ef000, maxevents: 0x1000, 
> timeout: 0x   
> ▒
>1.98%  epfd: 0x00b5, events: 0x7fca419ef000, maxevents: 0x1000, 
> timeout: 0x03e8   
> ▒
>0.04%  epfd: 0x0003, events: 0x2f6af77b9c00, maxevents: 0x0020, 
> timeout: 0x   
> ▒
>0.04%  epfd: 0x002b, events: 0x121ebf63ac00, maxevents: 0x0040, 
> timeout: 0x   
> ▒
>0.03%  epfd: 0x0026, events: 0x7f51f80019c0, maxevents: 0x0020, 
> timeout: 0x   
> ▒
>0.02%  epfd: 0x0003, events: 0x7fe4d80019d0, maxevents: 0x0020, 
> timeout: 0x
> {code}
> Running this time with perf record -ag for call traces:
> {code}
> # Children  Self   sys   usr  Trace output
> 
> #         
> 
> #
>  8.61% 8.61% 0.00% 8.61%  epfd: 0x00a7, events: 
> 0x7fca452d6000, maxevents: 0x1000, timeout: 0x
> |
> ---0x1000200af313
>|  
> --8.61%--0x7fca6117bdac
>   0x7fca60459804
>   epoll_wait
>  2.98% 2.98% 0.00% 2.98%  epfd: 0x00a7, events: 
> 0x7fca452d6000, maxevents: 0x1000, timeout: 0x03e8
> |
> ---0x1000200af313
>0x7fca6117b830
>0x7fca60459804
>epoll_wait
> {code}
> That looks like a lot of CPU used to wait for nothing. I'm not sure if pref 
> reports a per-CPU percentage or a per-system percentage, but that would be 
> still be 10% of the total CPU usage of Cassandra at the minimum.
> I went further and found the code of all that: We schedule a lot of 
> {{Message::Flusher}} with a deadline of 10 usec (5 per messages I think) but 
> netty+epoll only support timeouts above the milliseconds and will 

[jira] [Commented] (CASSANDRA-8119) More Expressive Consistency Levels

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171158#comment-16171158
 ] 

Kurt Greaves commented on CASSANDRA-8119:
-

Is it actually necessary to have the creation of CL's through CQL? I agree we 
need more from CL's. In the past I've needed LOCAL_ALL, ANY_QUORUM, and 
EACH_SERIAL. So this is undoubtedly a thing. But seems to me if CL's were an 
extendable class people could implement the CL's they need and make them 
publicly available. Might need some extra work to support the dc-wise stuff 
mentioned here but I think it's more than acceptable to expect people to build 
a jar with a specific CL they need and then add it to the classpath. If there 
are popular ones they could be candidates for adding to the codebase.

A UDF style thing seems a bit much imo, I don't really think they are a big 
enough use case to warrant the ability to define them on the fly... plus this 
also scares me a little bit.

> More Expressive Consistency Levels
> --
>
> Key: CASSANDRA-8119
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8119
> Project: Cassandra
>  Issue Type: New Feature
>  Components: CQL
>Reporter: Tyler Hobbs
> Fix For: 4.x
>
>
> For some multi-datacenter environments, the current set of consistency levels 
> are too restrictive.  For example, the following consistency requirements 
> cannot be expressed:
> * LOCAL_QUORUM in two specific DCs
> * LOCAL_QUORUM in the local DC plus LOCAL_QUORUM in at least one other DC
> * LOCAL_QUORUM in the local DC plus N remote replicas in any DC
> I propose that we add a new consistency level: CUSTOM.  In the v4 (or v5) 
> protocol, this would be accompanied by an additional map argument.  A map of 
> {DC: CL} or a map of {DC: int} is sufficient to cover the first example.  If 
> we accept a special keys to represent "any datacenter", the second case can 
> be handled.  A similar technique could be used for "any other nodes".
> I'm not in love with the special keys, so if anybody has ideas for something 
> more elegant, feel free to propose them.  The main idea is that we want to be 
> flexible enough to cover any reasonable consistency or durability 
> requirements.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org