[jira] [Comment Edited] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387326#comment-16387326
 ] 

Jaydeepkumar Chovatia edited comment on CASSANDRA-13740 at 3/6/18 5:47 AM:
---

Hi [~iamaleksey]

 A gentle ping. If you have some time then can you please help me close this?

 Jaydeep


was (Author: chovatia.jayd...@gmail.com):
Hi [~iamaleksey]

 

A gentle ping. If you have some time then can you please help me close this?

 

Jaydeep

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Hints
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387326#comment-16387326
 ] 

Jaydeepkumar Chovatia commented on CASSANDRA-13740:
---

Hi [~iamaleksey]

 

A gentle ping. If you have some time then can you please help me close this?

 

Jaydeep

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Hints
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387303#comment-16387303
 ] 

Jaydeepkumar Chovatia edited comment on CASSANDRA-12151 at 3/6/18 5:16 AM:
---

As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every 
single query will be audited once audit is enabled, can we make this selective 
to not audit all the queries but percentage of queries as people will have 
different requirements on audit query frequency?


was (Author: chovatia.jayd...@gmail.com):
As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every 
single query will be audited once audit is enabled, can we make this selective 
to not audit all the queries but percentage of queries as people will have 
different requirements for audit query frequency?

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387303#comment-16387303
 ] 

Jaydeepkumar Chovatia commented on CASSANDRA-12151:
---

As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every 
single query will be audited once audit is enabled, can we make this selective 
to not audit all the queries but percentage of queries as people will have 
different requirements for audit query frequency?

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387292#comment-16387292
 ] 

ZhaoYang commented on CASSANDRA-14292:
--

CASSANDRA-14194 was solving race condition with multiple mutations. This ticket 
is to solve performance for single mutation.

In 3.0 branch with 13987, each mutation will have to wait for full 
batch-sync-window(2ms) for commitlog to sync.

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang edited comment on CASSANDRA-14292 at 3/6/18 4:35 AM:
--

|[3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0]| 
unit running |dtest running|

Changes:

1. Set {{syncRequested}} to true in {{BatchCommitLogService.maybeWaitForSync, 
so batch CL will sync immediately when mutation comes}}

2. Added unit test to verify batch CL sync and shutdown immediately


was (Author: jasonstack):
|[3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0]| 
unit running |dtest running|

Changes:

1. Set {{syncRequested}} to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14099) LCS ordering of sstables by timestamp is inverted

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387237#comment-16387237
 ] 

Kurt Greaves commented on CASSANDRA-14099:
--

I'd say there's not really a need to create an "ascending" and "descending". 
Ascending can be acquired by simply calling 
{{maxTimestampComparator.reversed()}} and this should be satisfactory for 
anything that needs it.

I'd just change {{ageSortedSSTables}} to specify {{reversed()}} on the 
comparator. Would save us storing 2 copies of the comparator needlessly and 
changing the existing {{maxTimestampComparator}}.

> LCS ordering of sstables by timestamp is inverted
> -
>
> Key: CASSANDRA-14099
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14099
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jeff Jirsa
>Assignee: Vincent White
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> In CASSANDRA-14010 we discovered that CASSANDRA-13776 broke sstable ordering 
> by timestamp (inverted it accidentally). Investigating that revealed that the 
> comparator was expecting newest-to-oldest for read command, but LCS expects 
> oldest-to-newest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387232#comment-16387232
 ] 

Jason Brown commented on CASSANDRA-14292:
-

Please take a look to see if CASSANDRA-14194 (committed today) helps. I'll read 
this more thoroughly, as well.

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang edited comment on CASSANDRA-14292 at 3/6/18 2:17 AM:
--

|[3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0]| 
unit running |dtest running|

Changes:

1. Set {{syncRequested}} to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately


was (Author: jasonstack):
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] | 
unit running | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang edited comment on CASSANDRA-14292 at 3/6/18 2:13 AM:
--

| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] | 
unit running | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately


was (Author: jasonstack):
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0 |  
unit running] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang edited comment on CASSANDRA-14292 at 3/6/18 2:05 AM:
--

| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] |  
unit running] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately


was (Author: jasonstack):
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] |  
[unit|https://circleci.com/gh/jasonstack/cassandra/716] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang edited comment on CASSANDRA-14292 at 3/6/18 2:05 AM:
--

| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0 |  
unit running] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately


was (Author: jasonstack):
| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] |  
unit running] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-14292:
-
Status: Patch Available  (was: Open)

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387146#comment-16387146
 ] 

ZhaoYang commented on CASSANDRA-14292:
--

| [3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-14292-3.0] |  
[unit|https://circleci.com/gh/jasonstack/cassandra/716] | dtest running |

Changes:

1. Set `syncRequested` to true in {{BatchCommitLogService.maybeWaitForSync}}

2. Added unit test to verify batch CL sync and shutdown immediately

> Batch commitlog performance regression in 3.0.16
> 
>
> Key: CASSANDRA-14292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
> Project: Cassandra
>  Issue Type: Bug
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 3.0.x
>
>
> Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced 
> to disk right after mutation comes.
>  * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
>  * AbstractCommitlogService will continue and sync to disk
> After C-13987, it makes a branch for chain maker flush more frequently in 
> periodic mode. To make sure in batch mode CL still flushes immediately, it 
> added {{syncRequested}} flag.
>  Unfortunately, in 3.0 branch, this flag is not being set to true when 
> mutation is waiting.
> So in AbstractCommitlogService, it will not execute the CL sync branch until 
> it reaches sync window(2ms)..
> {code:java|title=AbstractCommitLogService.java}
> if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
> syncRequested)
> {
> // in this branch, we want to flush the commit log to disk
> syncRequested = false;
> commitLog.sync(shutdown, true);
> lastSyncedAt = pollStarted;
> syncComplete.signalAll();
> }
> else
> {
> // in this branch, just update the commit log sync headers
> commitLog.sync(false, false);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14292) Batch commitlog performance regression in 3.0.16

[ZhaoYang (JIRA)]

ZhaoYang created CASSANDRA-14292:


 Summary: Batch commitlog performance regression in 3.0.16
 Key: CASSANDRA-14292
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14292
 Project: Cassandra
  Issue Type: Bug
Reporter: ZhaoYang
Assignee: ZhaoYang
 Fix For: 3.0.x


Prior to CASSANDRA-13987, in batch commitlog mode, commitlog will be synced to 
disk right after mutation comes.
 * haveWork semaphore is released in BatchCommitLogService.maybeWaitForSync
 * AbstractCommitlogService will continue and sync to disk

After C-13987, it makes a branch for chain maker flush more frequently in 
periodic mode. To make sure in batch mode CL still flushes immediately, it 
added {{syncRequested}} flag.
 Unfortunately, in 3.0 branch, this flag is not being set to true when mutation 
is waiting.

So in AbstractCommitlogService, it will not execute the CL sync branch until it 
reaches sync window(2ms)..
{code:java|title=AbstractCommitLogService.java}
if (lastSyncedAt + syncIntervalMillis <= pollStarted || shutdown || 
syncRequested)
{
// in this branch, we want to flush the commit log to disk
syncRequested = false;
commitLog.sync(shutdown, true);
lastSyncedAt = pollStarted;
syncComplete.signalAll();
}
else
{
// in this branch, just update the commit log sync headers
commitLog.sync(false, false);
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14204) Nodetool garbagecollect AssertionError

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387134#comment-16387134
 ] 

Kurt Greaves commented on CASSANDRA-14204:
--

patch LGTM but could probably have a test that will ensure we never include 
unrepaired sstables when onlyPurgeRepairedTombstones is true.

> Nodetool garbagecollect AssertionError
> --
>
> Key: CASSANDRA-14204
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14204
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Vincent White
>Assignee: Vincent White
>Priority: Minor
> Fix For: 3.11.x, 4.x
>
>
> When manually running a garbage collection compaction across a table with 
> unrepaired sstables and only_purge_repaired_tombstones set to true an 
> assertion error is thrown. This is because the unrepaired sstables aren't 
> being removed from the transaction as they are filtered out in 
> filterSSTables().
> ||3.11||trunk||
> |[branch|https://github.com/vincewhite/cassandra/commit/e13c822736edd3df3403c02e8ef90816f158cde2]|[branch|https://github.com/vincewhite/cassandra/commit/cc8828576404e72504d9b334be85f84c90e77aa7]|
> The stacktrace:
> {noformat}
> -- StackTrace --
> java.lang.AssertionError
>   at 
> org.apache.cassandra.db.compaction.CompactionManager.parallelAllSSTableOperation(CompactionManager.java:339)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager.performGarbageCollection(CompactionManager.java:476)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.garbageCollect(ColumnFamilyStore.java:1579)
>   at 
> org.apache.cassandra.service.StorageService.garbageCollect(StorageService.java:3069)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
>   at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
>   at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
>   at 
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
>   at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>   at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
>   at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
>   at sun.rmi.transport.Transport$1.run(Transport.java:200)
>   at sun.rmi.transport.Transport$1.run(Transport.java:197)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
>   at 
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
>   at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
>   at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
>   at 
> 

[jira] [Commented] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387062#comment-16387062
 ] 

Kurt Greaves commented on CASSANDRA-11163:
--

Thanks for the review [~cnlwsu]. Created CASSANDRA-14291 as a follow up.

> Summaries are needlessly rebuilt when the BF FP ratio is changed
> 
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 12 00:03 ma-8-big-TOC.txt
> -rw-r--r-- 1 root root 14902 Feb 12 00:03 ma-8-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 12 00:03 ma-8-big-Statistics.db
> -rw-r--r-- 1 root root   1458631 Feb 12 00:03 ma-8-big-Index.db
> -rw-r--r-- 1 root root 10808 Feb 12 00:03 ma-8-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 12 00:03 ma-8-big-Digest.crc32
> -rw-r--r-- 1 root root  19660275 Feb 12 00:03 ma-8-big-Data.db
> 

[jira] [Created] (CASSANDRA-14291) Nodetool command to recreate SSTable components

[Kurt Greaves (JIRA)]

Kurt Greaves created CASSANDRA-14291:


 Summary: Nodetool command to recreate SSTable components
 Key: CASSANDRA-14291
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14291
 Project: Cassandra
  Issue Type: Improvement
Reporter: Kurt Greaves


Need a JMX/Nodetool command to recreate components for SSTables without 
re-writing the data files.

Possible implementation idea:
Create a {{nodetool (recreate|regen)component}} command that would enable you 
to recreate  specific components of an SSTable, and also allow specifying 
SSTables or columnfamilies.
I'd say a flag for a list of components and a flag for SSTables with 
keyspace.columnfamilies as positional arguments would work

Alternatively this could become part of upgradesstables, but would likely make 
that command a bit bloated.

Background:
In CASSANDRA-11163 we changed it so summaries and bloomfilters were not 
regenerated or persisted on startup. This means we would rely on 
compactions/upgrades to regenerate the bloomfilter (or other components) after 
a configuration change. While this works, it's pretty inefficient on large 
tables just because you changed the bloomfilter size or summary chunk sizes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-11163:
-
Status: Ready to Commit  (was: Patch Available)

> Summaries are needlessly rebuilt when the BF FP ratio is changed
> 
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 12 00:03 ma-8-big-TOC.txt
> -rw-r--r-- 1 root root 14902 Feb 12 00:03 ma-8-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 12 00:03 ma-8-big-Statistics.db
> -rw-r--r-- 1 root root   1458631 Feb 12 00:03 ma-8-big-Index.db
> -rw-r--r-- 1 root root 10808 Feb 12 00:03 ma-8-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 12 00:03 ma-8-big-Digest.crc32
> -rw-r--r-- 1 root root  19660275 Feb 12 00:03 ma-8-big-Data.db
> -rw-r--r-- 1 root root  1204 Feb 12 00:03 ma-8-big-CRC.db
> 

[jira] [Commented] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387002#comment-16387002
 ] 

Kurt Greaves commented on CASSANDRA-11163:
--

Correct. As I noted previously 

bq. Only regenerate and persist the bloomfilter when it's missing - not when it 
has changed. This means we rely on compactions/upgradesstables to update the 
bloomfilter.
bq. There's definitely no reason to regenerate Summaries in this case, and as 
previously mentioned it's not great regenerating the bloomfilter unless you're 
going to persist it. I have added persistence for the bloomfilter (when it is 
regenerated), however I think it's a bad idea to do this on startup as it will 
likely be more time consuming than regenerating the summaries.

So the previous behaviour was to regenerate the BF in this case but *not* 
persist it on the next startup (this meant it would happen on every startup 
until compactions/upgrades had occured). The summaries would be regenerated and 
persisted on the next startup (pointlessly). Both of these things would slow 
startup time pretty significantly depending on how much data you had.

The new behaviour would be to avoid regenerating BF/Summaries at all on startup 
and instead rely on upgradesstables/compactions to update them. Summaries would 
only be recreated when necessary (when not loaded/corrupt/missing).

In trunk it might make sense to also add a nodetool command that will allow us 
to regenerate the bloomfilters/summaries/etc without re-writing the whole data 
file.

> Summaries are needlessly rebuilt when the BF FP ratio is changed
> 
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 

[jira] [Commented] (CASSANDRA-14128) Prune dead links from cassandra.apache.org

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387008#comment-16387008
 ] 

Kurt Greaves commented on CASSANDRA-14128:
--

It's not that big of a change and I think the only thing that really needs 
discussion is the third party page. Not really worth making another ticket. 
I'll make the amendments for the "in use" companies.

> Prune dead links from cassandra.apache.org
> --
>
> Key: CASSANDRA-14128
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14128
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: Jeff Jirsa
>Assignee: Kurt Greaves
>Priority: Trivial
>  Labels: lhf
> Attachments: 14128-site.patch
>
>
> Lots of dead links on the site, including the homepage. Should do some 
> pruning cleanup.
> Site repo is [here|https://svn.apache.org/repos/asf/cassandra/site/] (in svn) 
> for anyone who wishes to give this a shot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387009#comment-16387009
 ] 

Chris Lohfink commented on CASSANDRA-11163:
---

gotcha +1 on patch from me

I like idea of a {{nodetool recreatecomponent}} or something but that would be 
a different jira.

> Summaries are needlessly rebuilt when the BF FP ratio is changed
> 
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 12 00:03 ma-8-big-TOC.txt
> -rw-r--r-- 1 root root 14902 Feb 12 00:03 ma-8-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 12 00:03 ma-8-big-Statistics.db
> -rw-r--r-- 1 root root   1458631 Feb 12 00:03 ma-8-big-Index.db
> -rw-r--r-- 1 root root 10808 Feb 12 00:03 ma-8-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 12 00:03 ma-8-big-Digest.crc32
> -rw-r--r-- 1 root 

[jira] [Updated] (CASSANDRA-14194) Chain commit log marker potential performance regression in batch commit mode

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14194:

   Resolution: Fixed
 Reviewer: Ariel Weisberg
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   3.11.3
   3.0.17
   4.0
   Status: Resolved  (was: Ready to Commit)

committed as sha {{85fafd0c134cae5aa84133ad54d67f2dba28c953}}. Thanks for 
finding and reviewing, [~aweisberg]!

> Chain commit log marker potential performance regression in batch commit mode
> -
>
> Key: CASSANDRA-14194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14194
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Testing
>Reporter: Ariel Weisberg
>Assignee: Jason Brown
>Priority: Major
> Fix For: 4.0, 3.0.17, 3.11.3
>
> Attachments: jstack.txt
>
>
> CASSANDRA-13987 modified how the commit log sync thread works. I noticed that 
> cql3.ViewTest and ViewBuilderTaskTest have been timing out, but only in 
> CircleCI. When I jstack in CircleCI what I see is that the commit log writer 
> thread is parked and the threads trying to append to the commit log are 
> blocked waiting on it.
> I tested the commit before 13987 and it passed in CircleCI and then I tested 
> with 13987 and it timed out. I suspect this may be a general performance 
> regression and not just a CircleCI issue.
> And this is with write barriers disabled (sync thread doesn't actually sync) 
> so it wasn't blocked in the filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5f31bb41
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5f31bb41
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5f31bb41

Branch: refs/heads/trunk
Commit: 5f31bb4152986a1551edbc70981a33ff35d0dfc0
Parents: 551744a 7f02348
Author: Jason Brown 
Authored: Mon Mar 5 14:36:19 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:37:50 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5f31bb41/CHANGES.txt
--
diff --cc CHANGES.txt
index 72554c2,a4be758..5fefc02
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,210 -1,14 +1,211 @@@
 +4.0
 + * Use Murmur3 for validation compactions (CASSANDRA-14002)
 + * Comma at the end of the seed list is interpretated as localhost 
(CASSANDRA-14285)
 + * Refactor read executor and response resolver, abstract read repair 
(CASSANDRA-14058)
 + * Add optional startup delay to wait until peers are ready (CASSANDRA-13993)
 + * Add a few options to nodetool verify (CASSANDRA-14201)
 + * CVE-2017-5929 Security vulnerability and redefine default log rotation 
policy (CASSANDRA-14183)
 + * Use JVM default SSL validation algorithm instead of custom default 
(CASSANDRA-13259)
 + * Better document in code InetAddressAndPort usage post 7544, incorporate 
port into UUIDGen node (CASSANDRA-14226)
 + * Fix sstablemetadata date string for minLocalDeletionTime (CASSANDRA-14132)
 + * Make it possible to change neverPurgeTombstones during runtime 
(CASSANDRA-14214)
 + * Remove GossipDigestSynVerbHandler#doSort() (CASSANDRA-14174)
 + * Add nodetool clientlist (CASSANDRA-13665)
 + * Revert ProtocolVersion changes from CASSANDRA-7544 (CASSANDRA-14211)
 + * Non-disruptive seed node list reload (CASSANDRA-14190)
 + * Nodetool tablehistograms to print statics for all the tables 
(CASSANDRA-14185)
 + * Migrate dtests to use pytest and python3 (CASSANDRA-14134)
 + * Allow storage port to be configurable per node (CASSANDRA-7544)
 + * Make sub-range selection for non-frozen collections return null instead of 
empty (CASSANDRA-14182)
 + * BloomFilter serialization format should not change byte ordering 
(CASSANDRA-9067)
 + * Remove unused on-heap BloomFilter implementation (CASSANDRA-14152)
 + * Delete temp test files on exit (CASSANDRA-14153)
 + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867)
 + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066)
 + * Fix cassandra-stress startup failure (CASSANDRA-14106)
 + * Remove initialDirectories from CFS (CASSANDRA-13928)
 + * Fix trivial log format error (CASSANDRA-14015)
 + * Allow sstabledump to do a json object per partition (CASSANDRA-13848)
 + * Add option to optimise merkle tree comparison across replicas 
(CASSANDRA-3200)
 + * Remove unused and deprecated methods from AbstractCompactionStrategy 
(CASSANDRA-14081)
 + * Fix Distribution.average in cassandra-stress (CASSANDRA-14090)
 + * Support a means of logging all queries as they were invoked 
(CASSANDRA-13983)
 + * Presize collections (CASSANDRA-13760)
 + * Add GroupCommitLogService (CASSANDRA-13530)
 + * Parallelize initial materialized view build (CASSANDRA-12245)
 + * Fix flaky SecondaryIndexManagerTest.assert[Not]MarkedAsBuilt 
(CASSANDRA-13965)
 + * Make LWTs send resultset metadata on every request (CASSANDRA-13992)
 + * Fix flaky indexWithFailedInitializationIsNotQueryableAfterPartialRebuild 
(CASSANDRA-13963)
 + * Introduce leaf-only iterator (CASSANDRA-9988)
 + * Upgrade Guava to 23.3 and Airline to 0.8 (CASSANDRA-13997)
 + * Allow only one concurrent call to StatusLogger (CASSANDRA-12182)
 + * Refactoring to specialised functional interfaces (CASSANDRA-13982)
 + * Speculative retry should allow more friendly params (CASSANDRA-13876)
 + * Throw exception if we send/receive repair messages to incompatible nodes 
(CASSANDRA-13944)
 + * Replace usages of MessageDigest with Guava's Hasher (CASSANDRA-13291)
 + * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728)
 + * Fix some alerts raised by static analysis (CASSANDRA-13799)
 + * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593)
 + * Add result set metadata to prepared statement MD5 hash calculation 
(CASSANDRA-10786)
 + * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941)
 + * Expose recent histograms in JmxHistograms (CASSANDRA-13642)
 + * Fix buffer length comparison when 

[2/6] cassandra git commit: Chain commit log marker potential performance regression in batch commit mode

[jasobrown]

Chain commit log marker potential performance regression in batch commit mode

patch by jasobrown; reviewed by Ariel Weisberg for CASSANDRA-14194


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85fafd0c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85fafd0c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85fafd0c

Branch: refs/heads/cassandra-3.11
Commit: 85fafd0c134cae5aa84133ad54d67f2dba28c953
Parents: f88ec93
Author: Jason Brown 
Authored: Tue Feb 20 14:16:59 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:24:18 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5599906..9734507 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 829530d..f3cfd29 100644
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
@@ -37,7 +37,7 @@ public abstract class AbstractCommitLogService
  */
 static final long DEFAULT_MARKER_INTERVAL_MILLIS = 100;
 
-private Thread thread;
+private volatile Thread thread;
 private volatile boolean shutdown = false;
 
 // all Allocations written before this time will be synced
@@ -168,8 +168,8 @@ public abstract class AbstractCommitLogService
 if (lastSyncedAt + syncIntervalMillis <= pollStarted || 
shutdown || syncRequested)
 {
 // in this branch, we want to flush the commit log to disk
-commitLog.sync(shutdown, true);
 syncRequested = false;
+commitLog.sync(shutdown, true);
 lastSyncedAt = pollStarted;
 syncComplete.signalAll();
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 8834c8c..b803d88 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -337,11 +337,12 @@ public abstract class CommitLogSegment
 {
 flush(startMarker, sectionEnd);
 lastSyncedOffset = lastMarkerOffset = nextMarker;
-}
 
-if (close)
-internalClose();
-syncComplete.signalAll();
+if (close)
+internalClose();
+
+syncComplete.signalAll();
+}
 }
 
 protected static void writeSyncMarker(long id, ByteBuffer buffer, int 
offset, int filePos, int nextMarker)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f02348e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f02348e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f02348e

Branch: refs/heads/cassandra-3.11
Commit: 7f02348ecf5f611c8f8cc793425ca8cf3f9099c9
Parents: d6982cd 85fafd0
Author: Jason Brown 
Authored: Mon Mar 5 14:34:21 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:35:32 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f02348e/CHANGES.txt
--
diff --cc CHANGES.txt
index 163d787,9734507..a4be758
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,44 -1,19 +1,45 @@@
 -3.0.17
 +3.11.3
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
   * Fully utilise specified compaction threads (CASSANDRA-14210)
   * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 -Merged from 2.1:
 - * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  Merged from 2.2:
   * Backport circleci yaml (CASSANDRA-14240)
 +Merged from 2.1:
 + * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  
 -3.0.16
 - * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)
 - * Add MinGW uname check to start scripts (CASSANDRA-12940)
 - * Protect against overflow of local expiration time (CASSANDRA-14092)
 +
 +3.11.2
 + * Fix ReadCommandTest (CASSANDRA-14234)
 + * Remove trailing period from latency reports at keyspace level 
(CASSANDRA-14233)
 + * Backport CASSANDRA-13080: Use new token allocation for non bootstrap case 
as well (CASSANDRA-14212)
 + * Remove dependencies on JVM internal classes from JMXServerUtils 
(CASSANDRA-14173) 
 + * Add DEFAULT, UNSET, MBEAN and MBEANS to `ReservedKeywords` 
(CASSANDRA-14205)
 + * Add Unittest for schema migration fix (CASSANDRA-14140)
 + * Print correct snitch info from nodetool describecluster (CASSANDRA-13528)
 + * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
 + * Enable CDC unittest (CASSANDRA-14141)
 + * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
 + * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 + * Correctly count range tombstones in traces and tombstone thresholds 
(CASSANDRA-8527)
 +Merged from 3.0:
 + * Add MinGW uname check to start scripts (CASSANDRA-12840)
   * Use the correct digest file and reload sstable metadata in nodetool verify 
(CASSANDRA-14217)
   * Handle failure when mutating repaired status in Verifier (CASSANDRA-13933)
 - * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
   * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f02348e/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 4571b54,f3cfd29..7c5d300
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ 

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f02348e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f02348e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f02348e

Branch: refs/heads/trunk
Commit: 7f02348ecf5f611c8f8cc793425ca8cf3f9099c9
Parents: d6982cd 85fafd0
Author: Jason Brown 
Authored: Mon Mar 5 14:34:21 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:35:32 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f02348e/CHANGES.txt
--
diff --cc CHANGES.txt
index 163d787,9734507..a4be758
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,44 -1,19 +1,45 @@@
 -3.0.17
 +3.11.3
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
   * Fully utilise specified compaction threads (CASSANDRA-14210)
   * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 -Merged from 2.1:
 - * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  Merged from 2.2:
   * Backport circleci yaml (CASSANDRA-14240)
 +Merged from 2.1:
 + * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  
 -3.0.16
 - * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)
 - * Add MinGW uname check to start scripts (CASSANDRA-12940)
 - * Protect against overflow of local expiration time (CASSANDRA-14092)
 +
 +3.11.2
 + * Fix ReadCommandTest (CASSANDRA-14234)
 + * Remove trailing period from latency reports at keyspace level 
(CASSANDRA-14233)
 + * Backport CASSANDRA-13080: Use new token allocation for non bootstrap case 
as well (CASSANDRA-14212)
 + * Remove dependencies on JVM internal classes from JMXServerUtils 
(CASSANDRA-14173) 
 + * Add DEFAULT, UNSET, MBEAN and MBEANS to `ReservedKeywords` 
(CASSANDRA-14205)
 + * Add Unittest for schema migration fix (CASSANDRA-14140)
 + * Print correct snitch info from nodetool describecluster (CASSANDRA-13528)
 + * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
 + * Enable CDC unittest (CASSANDRA-14141)
 + * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
 + * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 + * Correctly count range tombstones in traces and tombstone thresholds 
(CASSANDRA-8527)
 +Merged from 3.0:
 + * Add MinGW uname check to start scripts (CASSANDRA-12840)
   * Use the correct digest file and reload sstable metadata in nodetool verify 
(CASSANDRA-14217)
   * Handle failure when mutating repaired status in Verifier (CASSANDRA-13933)
 - * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
   * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f02348e/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 4571b54,f3cfd29..7c5d300
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ 

[3/6] cassandra git commit: Chain commit log marker potential performance regression in batch commit mode

[jasobrown]

Chain commit log marker potential performance regression in batch commit mode

patch by jasobrown; reviewed by Ariel Weisberg for CASSANDRA-14194


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85fafd0c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85fafd0c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85fafd0c

Branch: refs/heads/trunk
Commit: 85fafd0c134cae5aa84133ad54d67f2dba28c953
Parents: f88ec93
Author: Jason Brown 
Authored: Tue Feb 20 14:16:59 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:24:18 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5599906..9734507 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 829530d..f3cfd29 100644
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
@@ -37,7 +37,7 @@ public abstract class AbstractCommitLogService
  */
 static final long DEFAULT_MARKER_INTERVAL_MILLIS = 100;
 
-private Thread thread;
+private volatile Thread thread;
 private volatile boolean shutdown = false;
 
 // all Allocations written before this time will be synced
@@ -168,8 +168,8 @@ public abstract class AbstractCommitLogService
 if (lastSyncedAt + syncIntervalMillis <= pollStarted || 
shutdown || syncRequested)
 {
 // in this branch, we want to flush the commit log to disk
-commitLog.sync(shutdown, true);
 syncRequested = false;
+commitLog.sync(shutdown, true);
 lastSyncedAt = pollStarted;
 syncComplete.signalAll();
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 8834c8c..b803d88 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -337,11 +337,12 @@ public abstract class CommitLogSegment
 {
 flush(startMarker, sectionEnd);
 lastSyncedOffset = lastMarkerOffset = nextMarker;
-}
 
-if (close)
-internalClose();
-syncComplete.signalAll();
+if (close)
+internalClose();
+
+syncComplete.signalAll();
+}
 }
 
 protected static void writeSyncMarker(long id, ByteBuffer buffer, int 
offset, int filePos, int nextMarker)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Chain commit log marker potential performance regression in batch commit mode

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 f88ec9357 -> 85fafd0c1
  refs/heads/cassandra-3.11 d6982cd22 -> 7f02348ec
  refs/heads/trunk 551744a99 -> 5f31bb415


Chain commit log marker potential performance regression in batch commit mode

patch by jasobrown; reviewed by Ariel Weisberg for CASSANDRA-14194


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85fafd0c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85fafd0c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85fafd0c

Branch: refs/heads/cassandra-3.0
Commit: 85fafd0c134cae5aa84133ad54d67f2dba28c953
Parents: f88ec93
Author: Jason Brown 
Authored: Tue Feb 20 14:16:59 2018 -0800
Committer: Jason Brown 
Committed: Mon Mar 5 14:24:18 2018 -0800

--
 CHANGES.txt | 1 +
 .../cassandra/db/commitlog/AbstractCommitLogService.java| 4 ++--
 .../org/apache/cassandra/db/commitlog/CommitLogSegment.java | 9 +
 3 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5599906..9734507 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 829530d..f3cfd29 100644
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
@@ -37,7 +37,7 @@ public abstract class AbstractCommitLogService
  */
 static final long DEFAULT_MARKER_INTERVAL_MILLIS = 100;
 
-private Thread thread;
+private volatile Thread thread;
 private volatile boolean shutdown = false;
 
 // all Allocations written before this time will be synced
@@ -168,8 +168,8 @@ public abstract class AbstractCommitLogService
 if (lastSyncedAt + syncIntervalMillis <= pollStarted || 
shutdown || syncRequested)
 {
 // in this branch, we want to flush the commit log to disk
-commitLog.sync(shutdown, true);
 syncRequested = false;
+commitLog.sync(shutdown, true);
 lastSyncedAt = pollStarted;
 syncComplete.signalAll();
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85fafd0c/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 8834c8c..b803d88 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -337,11 +337,12 @@ public abstract class CommitLogSegment
 {
 flush(startMarker, sectionEnd);
 lastSyncedOffset = lastMarkerOffset = nextMarker;
-}
 
-if (close)
-internalClose();
-syncComplete.signalAll();
+if (close)
+internalClose();
+
+syncComplete.signalAll();
+}
 }
 
 protected static void writeSyncMarker(long id, ByteBuffer buffer, int 
offset, int filePos, int nextMarker)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-7423) Allow updating individual subfields of UDT

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386662#comment-16386662
 ] 

Bhanu M. Gandikota commented on CASSANDRA-7423:
---

Hi I got the following error stack during "nodetool upgrades stables" [from 
2.2.5 to 3.11.1]. What do I need to do ?

 

-bash-4.2$ nodetool upgradesstables

WARN  11:28:28,430 Small cdc volume detected at /cdc_raw; setting 
cdc_total_space_in_mb to 1982.  You can override this in cassandra.yaml

error: null

-- StackTrace --

java.lang.AssertionError

at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:43)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:495)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)

at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:176)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:49)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:59)

at 
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:384)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:122)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:113)

at 
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:466)

at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:163)

at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92)

at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)

at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)

at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)

at 
org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)

at 
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)

at java.lang.Thread.run(Thread.java:745)

 

> Allow updating individual subfields of UDT
> --
>
> Key: CASSANDRA-7423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7423
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Tupshin Harper
>Assignee: Tyler Hobbs
>Priority: Major
>  Labels: client-impacting, cql, docs-impacting
> Fix For: 3.6
>
>
> Since user defined types were implemented in CASSANDRA-5590 as blobs (you 
> have to rewrite the entire type in order to make any modifications), they 
> can't be safely used without LWT for any operation that wants to modify a 
> subset of the UDT's fields by any client process that is not authoritative 
> for the entire blob. 
> When trying to use UDTs to model complex records (particularly with nesting), 
> this is not an exceptional circumstance, this is the totally expected normal 
> situation. 
> The use of UDTs for anything non-trivial is harmful to either performance or 
> consistency or both.
> edit: to clarify, i believe that most potential uses of UDTs should be 
> considered anti-patterns until/unless we have field-level r/w access to 
> individual elements of the UDT, with individual timestamps and standard LWW 
> semantics



--
This message was sent by Atlassian JIRA

[jira] [Commented] (CASSANDRA-14194) Chain commit log marker potential performance regression in batch commit mode

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386664#comment-16386664
 ] 

Jason Brown commented on CASSANDRA-14194:
-

bq.  #2 This won't happen because LockSupport.unpark() will cause it to wake 
immediately I believe?

Reading the docs you linked, this sounds correct. I will remove that change on 
commit. (thanks! this was a good pointer)

> Chain commit log marker potential performance regression in batch commit mode
> -
>
> Key: CASSANDRA-14194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14194
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Testing
>Reporter: Ariel Weisberg
>Assignee: Jason Brown
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: jstack.txt
>
>
> CASSANDRA-13987 modified how the commit log sync thread works. I noticed that 
> cql3.ViewTest and ViewBuilderTaskTest have been timing out, but only in 
> CircleCI. When I jstack in CircleCI what I see is that the commit log writer 
> thread is parked and the threads trying to append to the commit log are 
> blocked waiting on it.
> I tested the commit before 13987 and it passed in CircleCI and then I tested 
> with 13987 and it timed out. I suspect this may be a general performance 
> regression and not just a CircleCI issue.
> And this is with write barriers disabled (sync thread doesn't actually sync) 
> so it wasn't blocked in the filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386644#comment-16386644
 ] 

Bhanu M. Gandikota edited comment on CASSANDRA-14113 at 3/5/18 7:49 PM:


FYI, here is my JDK version:

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

 
{code:java}
 {code}


was (Author: bhanu.m.gandikota):
FYI, here is my JDK version:

 \{code:sql}

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

 

 

{code}

> AssertionError while trying to upgrade 2.2.11 -> 3.11.1
> ---
>
> Key: CASSANDRA-14113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14113
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Tables have been created in 2.2.11 using thrift and have 
> supercolumns
>Reporter: Guillaume Herail
>Priority: Major
>  Labels: supercolumns
> Attachments: data.tar.gz
>
>
> We're trying to upgrade a test cluster from Cassandra 2.2.11 to Cassandra 
> 3.11.1. The tables have been created using thrift and have supercolumns. When 
> I try to run {{nodetool upgradesstables}} I get the following:
> {noformat}error: null
> -- StackTrace --
> java.lang.AssertionError
>   at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:42)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:498)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:188)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:140)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:484)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
>   at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:75)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
>   at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
>   at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Comment Edited] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386644#comment-16386644
 ] 

Bhanu M. Gandikota edited comment on CASSANDRA-14113 at 3/5/18 7:49 PM:


FYI, here is my JDK version:

 \{code:sql}

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

 

 

{code}


was (Author: bhanu.m.gandikota):
FYI, here is my JDK version:

 \{code:sql}

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

{code}

> AssertionError while trying to upgrade 2.2.11 -> 3.11.1
> ---
>
> Key: CASSANDRA-14113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14113
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Tables have been created in 2.2.11 using thrift and have 
> supercolumns
>Reporter: Guillaume Herail
>Priority: Major
>  Labels: supercolumns
> Attachments: data.tar.gz
>
>
> We're trying to upgrade a test cluster from Cassandra 2.2.11 to Cassandra 
> 3.11.1. The tables have been created using thrift and have supercolumns. When 
> I try to run {{nodetool upgradesstables}} I get the following:
> {noformat}error: null
> -- StackTrace --
> java.lang.AssertionError
>   at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:42)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:498)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:188)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:140)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:484)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
>   at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:75)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
>   at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
>   at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Comment Edited] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386644#comment-16386644
 ] 

Bhanu M. Gandikota edited comment on CASSANDRA-14113 at 3/5/18 7:48 PM:


FYI, here is my JDK version:

 \{code:sql}

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

{code}


was (Author: bhanu.m.gandikota):
FYI, here is my JDK version:

 

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

> AssertionError while trying to upgrade 2.2.11 -> 3.11.1
> ---
>
> Key: CASSANDRA-14113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14113
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Tables have been created in 2.2.11 using thrift and have 
> supercolumns
>Reporter: Guillaume Herail
>Priority: Major
>  Labels: supercolumns
> Attachments: data.tar.gz
>
>
> We're trying to upgrade a test cluster from Cassandra 2.2.11 to Cassandra 
> 3.11.1. The tables have been created using thrift and have supercolumns. When 
> I try to run {{nodetool upgradesstables}} I get the following:
> {noformat}error: null
> -- StackTrace --
> java.lang.AssertionError
>   at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:42)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:498)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:188)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:140)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:484)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
>   at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:75)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
>   at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
>   at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Comment Edited] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386637#comment-16386637
 ] 

Bhanu M. Gandikota edited comment on CASSANDRA-14113 at 3/5/18 7:48 PM:


Here there, I've been receiving the exact stack errors when I tried to upgrade 
from 2.2.5 to 3.11.1 using "nodetool upgradesstables". 

I'd appreciate if a solution is provided to this issue. 

Just wanted to let you know that I don't see any other errors/warnings in 
system.log or in debug.log. Just the upgrades stables fails after running for a 
while. 

{code:sql}

$ nodetool upgradesstables

WARN  11:28:28,430 Small cdc volume detected at /cdc_raw; setting 
cdc_total_space_in_mb to 1982.  You can override this in cassandra.yaml

error: null

– StackTrace –

java.lang.AssertionError

at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:43)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:495)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)

at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:176)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:49)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:59)

at 
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:384)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:122)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:113)

at 
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:466)

at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:163)

at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92)

at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)

at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)

at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)

at 
org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)

at 
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)

at java.lang.Thread.run(Thread.java:745)
{code:java}
 {code}


was (Author: bhanu.m.gandikota):
Here there, I've been receiving the exact stack errors when I tried to upgrade 
from 2.2.5 to 3.11.1 using "nodetool upgradesstables". 

I'd appreciate if a solution is provided to this issue. 

Just wanted to let you know that I don't see any other errors/warnings in 
system.log or in debug.log. Just the upgrades stables fails after running for a 
while. 

$ nodetool upgradesstables

WARN  11:28:28,430 Small cdc volume detected at /cdc_raw; setting 
cdc_total_space_in_mb to 1982.  You can override this in cassandra.yaml

error: null

-- StackTrace --

java.lang.AssertionError

at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:43)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:495)

at 

[jira] [Commented] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386644#comment-16386644
 ] 

Bhanu M. Gandikota commented on CASSANDRA-14113:


FYI, here is my JDK version:

 

-bash-4.2$ java -version

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 

> AssertionError while trying to upgrade 2.2.11 -> 3.11.1
> ---
>
> Key: CASSANDRA-14113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14113
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Tables have been created in 2.2.11 using thrift and have 
> supercolumns
>Reporter: Guillaume Herail
>Priority: Major
>  Labels: supercolumns
> Attachments: data.tar.gz
>
>
> We're trying to upgrade a test cluster from Cassandra 2.2.11 to Cassandra 
> 3.11.1. The tables have been created using thrift and have supercolumns. When 
> I try to run {{nodetool upgradesstables}} I get the following:
> {noformat}error: null
> -- StackTrace --
> java.lang.AssertionError
>   at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:42)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:498)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)
>   at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:188)
>   at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:140)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:484)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499)
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359)
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>   at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
>   at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:74)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:75)
>   at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
>   at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
>   at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>   at 

[jira] [Commented] (CASSANDRA-14113) AssertionError while trying to upgrade 2.2.11 -> 3.11.1

[Bhanu M. Gandikota (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386637#comment-16386637
 ] 

Bhanu M. Gandikota commented on CASSANDRA-14113:


Here there, I've been receiving the exact stack errors when I tried to upgrade 
from 2.2.5 to 3.11.1 using "nodetool upgradesstables". 

I'd appreciate if a solution is provided to this issue. 

Just wanted to let you know that I don't see any other errors/warnings in 
system.log or in debug.log. Just the upgrades stables fails after running for a 
while. 

$ nodetool upgradesstables

WARN  11:28:28,430 Small cdc volume detected at /cdc_raw; setting 
cdc_total_space_in_mb to 1982.  You can override this in cassandra.yaml

error: null

-- StackTrace --

java.lang.AssertionError

at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:43)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)

at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:495)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:472)

at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:306)

at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:176)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:49)

at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:59)

at 
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:384)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)

at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:122)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:113)

at 
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:466)

at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)

at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:163)

at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92)

at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:233)

at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:196)

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)

at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)

at 
org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:428)

at 
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:315)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)

at java.lang.Thread.run(Thread.java:745)

> AssertionError while trying to upgrade 2.2.11 -> 3.11.1
> ---
>
> Key: CASSANDRA-14113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14113
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Tables have been created in 2.2.11 using thrift and have 
> supercolumns
>Reporter: Guillaume Herail
>Priority: Major
>  Labels: supercolumns
> Attachments: data.tar.gz
>
>
> We're trying to upgrade a test cluster from Cassandra 2.2.11 to Cassandra 
> 3.11.1. The tables have been created using thrift and have supercolumns. When 
> I try to run {{nodetool upgradesstables}} I get the following:
> {noformat}error: null
> -- StackTrace --
> java.lang.AssertionError
>   at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:42)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1242)
>   at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1185)
>   at 
> 

[jira] [Created] (CASSANDRA-14290) When is 4.0 release planned

[Maitrayee (JIRA)]

Maitrayee created CASSANDRA-14290:
-

 Summary: When is 4.0 release planned
 Key: CASSANDRA-14290
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14290
 Project: Cassandra
  Issue Type: Task
Reporter: Maitrayee






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14194) Chain commit log marker potential performance regression in batch commit mode

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14194:
---
Status: Ready to Commit  (was: Patch Available)

> Chain commit log marker potential performance regression in batch commit mode
> -
>
> Key: CASSANDRA-14194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14194
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Testing
>Reporter: Ariel Weisberg
>Assignee: Jason Brown
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: jstack.txt
>
>
> CASSANDRA-13987 modified how the commit log sync thread works. I noticed that 
> cql3.ViewTest and ViewBuilderTaskTest have been timing out, but only in 
> CircleCI. When I jstack in CircleCI what I see is that the commit log writer 
> thread is parked and the threads trying to append to the commit log are 
> blocked waiting on it.
> I tested the commit before 13987 and it passed in CircleCI and then I tested 
> with 13987 and it timed out. I suspect this may be a general performance 
> regression and not just a CircleCI issue.
> And this is with write barriers disabled (sync thread doesn't actually sync) 
> so it wasn't blocked in the filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14194) Chain commit log marker potential performance regression in batch commit mode

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386485#comment-16386485
 ] 

Ariel Weisberg commented on CASSANDRA-14194:


RE #1 I think this is a correct fix. Totally makes sense that clobbering the 
more work indicator after already starting to collect the next batch of work is 
going to cause this issue.

RE #2 This won't happen because LockSupport.unpark() will cause it to wake 
immediately I believe? 
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/LockSupport.html#unpark(java.lang.Thread)

RE #3  looks good.

RE #4 should be fine. It's only read when doing hideously expensive operations 
that dwarf the cost of a volatile read anyways.

+1

> Chain commit log marker potential performance regression in batch commit mode
> -
>
> Key: CASSANDRA-14194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14194
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Testing
>Reporter: Ariel Weisberg
>Assignee: Jason Brown
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: jstack.txt
>
>
> CASSANDRA-13987 modified how the commit log sync thread works. I noticed that 
> cql3.ViewTest and ViewBuilderTaskTest have been timing out, but only in 
> CircleCI. When I jstack in CircleCI what I see is that the commit log writer 
> thread is parked and the threads trying to append to the commit log are 
> blocked waiting on it.
> I tested the commit before 13987 and it passed in CircleCI and then I tested 
> with 13987 and it timed out. I suspect this may be a general performance 
> regression and not just a CircleCI issue.
> And this is with write barriers disabled (sync thread doesn't actually sync) 
> so it wasn't blocked in the filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14275) Cassandra Driver should send identification information to Server

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386477#comment-16386477
 ] 

Aleksey Yeschenko commented on CASSANDRA-14275:
---

Minor stuff: for better parse-ability, should alter the way you output 
{{clientstats}} output.

Rename 'Driver Name' into 'Driver-Name' and 'Driver Version' into 
'Driver-Version', and instead of empty strings, use {{"null"}} or 
{{"undefined"}} for name and version. Reason is that {{TableBuilder}} uses 
spaces as separator.

> Cassandra Driver should send identification information to Server
> -
>
> Key: CASSANDRA-14275
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14275
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.x
>
>
> Currently there doesn't seem to be any way to readily identify the driver 
> that clients are using to connect to Cassandra. Add the capability of 
> identifying the driver through metadata information much like how HTTP 
> Clients identify themselves through User-Agent HTTP header. This is useful 
> for debugging in large deployments where clients tend to use different 
> drivers, wrappers and language bindings to connect to Cassandra. This can 
> help surface issues as well as help detect clients which are using older or 
> unsupported drivers.
> The identification information should be a string that unambiguously 
> identifies the driver. It should include information such as the name of the 
> driver, it's version, CQL version, Platform (Linux, macOS, Windows, etc.) and 
> architecture (x86, x86_64).
> We should surface this information in `nodetool clientstats` command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14275) Cassandra Driver should send identification information to Server

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-14275:
--
Status: Open  (was: Patch Available)

> Cassandra Driver should send identification information to Server
> -
>
> Key: CASSANDRA-14275
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14275
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.x
>
>
> Currently there doesn't seem to be any way to readily identify the driver 
> that clients are using to connect to Cassandra. Add the capability of 
> identifying the driver through metadata information much like how HTTP 
> Clients identify themselves through User-Agent HTTP header. This is useful 
> for debugging in large deployments where clients tend to use different 
> drivers, wrappers and language bindings to connect to Cassandra. This can 
> help surface issues as well as help detect clients which are using older or 
> unsupported drivers.
> The identification information should be a string that unambiguously 
> identifies the driver. It should include information such as the name of the 
> driver, it's version, CQL version, Platform (Linux, macOS, Windows, etc.) and 
> architecture (x86, x86_64).
> We should surface this information in `nodetool clientstats` command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14275) Cassandra Driver should send identification information to Server

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386419#comment-16386419
 ] 

Aleksey Yeschenko commented on CASSANDRA-14275:
---

Hey o.

It's an ok start, but we can make it better:

1. {{ClientState()}} constructor doesn't need {{this.driverName = null;}} or 
{{this.driverVersion = null;}} assignments. You were probably just mimicking 
the other assignments in it, but you shouldn't have. They are there because 
{{isInternal}} and {{remoteAddress}} fields are {{final}}. {{driverName}} and 
{{driverVersion}} aren't, and will be set to null anyway.
2. Likewise, in {{ClientState(InetSocketAddress remoteAddress)}} constructor, 
{{this.driverName = null;}} assignment is not needed.
3. {{ClientState forExternalCalls(SocketAddress remoteAddress, String 
driverName, String driverVersion)}} method is not used by anything, and neither 
do we need {{ClientState(InetSocketAddress remoteAddress, String driverName, 
String driverVersion)}} constructor. You might be tempted to create them just 
because we might need them one day, but we try to generally avoid that. So I'd 
get rid of them both.
4. {{setDriverInfo(String driverName, String driverVersion)}} should probably 
be split into two methods, one for driver name and another for driver version. 
See below for reason why.
5. You shouldn't put the {{STARTUP}} handling code into {{Message}} class. What 
you did belongs to {{StartupMessage.execute()}} method.
6. {{"DRIVER_NAME"}} and {{"DRIVER_VERSION"}} should be constantized once in 
{{StartupMessage}}, like the rest of the options.
7. It should be possible to only set the driver name, but we shouldn't set 
driver version if name is not set.
8. I would prefer if you made {{getDriverVersion()}} and {{getDriverName()}} 
methods in {{ClientState}} return an {{Optional}} on read instead of 
using empty strings on write.

This is all for now, though I might have more in the second round. If you don't 
mind, can you squash all the existing comments into one, and address review 
feedback in separate commits on top of the squashed one?

Cheers (:

> Cassandra Driver should send identification information to Server
> -
>
> Key: CASSANDRA-14275
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14275
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.x
>
>
> Currently there doesn't seem to be any way to readily identify the driver 
> that clients are using to connect to Cassandra. Add the capability of 
> identifying the driver through metadata information much like how HTTP 
> Clients identify themselves through User-Agent HTTP header. This is useful 
> for debugging in large deployments where clients tend to use different 
> drivers, wrappers and language bindings to connect to Cassandra. This can 
> help surface issues as well as help detect clients which are using older or 
> unsupported drivers.
> The identification information should be a string that unambiguously 
> identifies the driver. It should include information such as the name of the 
> driver, it's version, CQL version, Platform (Linux, macOS, Windows, etc.) and 
> architecture (x86, x86_64).
> We should surface this information in `nodetool clientstats` command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14070) Add new method for returning list of primary/clustering key values

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386382#comment-16386382
 ] 

Aleksey Yeschenko commented on CASSANDRA-14070:
---

This doesn't look right to me. I would prefer if you stuck this into your 
Trigger code, as this is way too specific of a use case to be useful to a 
broader developer base. Sorry.

> Add new method for returning list of primary/clustering key values
> --
>
> Key: CASSANDRA-14070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14070
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Himani Arora
>Assignee: Himani Arora
>Priority: Minor
> Fix For: 4.x
>
>
> Add a method to return a list of primary/clustering key values so that it 
> will be easier to process data. Currently, we are getting a string 
> concatenated with either colon (: ) or comma (,) which makes it quite 
> difficult to fetch one single key value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386338#comment-16386338
 ] 

Chris Lohfink commented on CASSANDRA-11163:
---

Doesnt this now miss case where under normal (not offline) run and they change 
bf ratio it no longer recreates them? {{(validation.bloomFilterFPChance != 
metadata().params.bloomFilterFpChance)}}. With this it will just load the old 
bf so theres no way to change it.

> Summaries are needlessly rebuilt when the BF FP ratio is changed
> 
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr 
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root  4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root   2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root  35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root  2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root   2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root  35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root  2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root  10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root  8508 Feb 11 23:50 ma-5-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 12 00:03 ma-8-big-TOC.txt
> -rw-r--r-- 1 root root 14902 Feb 12 00:03 ma-8-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 12 00:03 ma-8-big-Statistics.db
> -rw-r--r-- 1 root root   1458631 Feb 12 00:03 ma-8-big-Index.db
> -rw-r--r-- 1 

[jira] [Commented] (CASSANDRA-5836) Seed nodes should be able to bootstrap without manual intervention

[Oleksandr Shulgin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386312#comment-16386312
 ] 

Oleksandr Shulgin commented on CASSANDRA-5836:
--

{quote}Don't assume that a new cluster is empty or that it's safe to assume 
that the first ever seed node has no data. If authentication is enabled, a 
default user is created. It would be very surprising for users ("can't login 
with cassandra anymore!") and actually a real security issue, if that user is 
recreated (if it was dropped) or the password changed back to the 
default.{quote}

I indeed remember myself seeing the default superuser created again due to a 
node restart on a cluster where it was dropped before that, but I'm not sure I 
fully understand the mechanism.  I might have forgotten some important detail, 
but I think it was related to setting up a secondary DC.  Unfortunately, I 
don't think it was trivially reproducible.  [~snazy], do you have more details 
about this issue?


> Seed nodes should be able to bootstrap without manual intervention
> --
>
> Key: CASSANDRA-5836
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5836
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Bill Hathaway
>Priority: Minor
>
> The current logic doesn't allow a seed node to be bootstrapped.  If a user 
> wants to bootstrap a node configured as a seed (for example to replace a seed 
> node via replace_token), they first need to remove the node's own IP from the 
> seed list, and then start the bootstrap process.  This seems like an 
> unnecessary step since a node never uses itself as a seed.
> I think it would be a better experience if the logic was changed to allow a 
> seed node to bootstrap without manual intervention when there are other seed 
> nodes up in a ring.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386288#comment-16386288
 ] 

Ariel Weisberg commented on CASSANDRA-12151:


I think logging more things to more formats, and making what is logged more 
configurable are great additions. I'm -1 on having two approaches to logging 
database access to a file that are redundant.

I don't really care what implementation we go with just that there aren't two. 
You could completely remove the existing FQL since we haven't shipped it and 
replace it with something with a similar high performance asynchronous binary 
logging option as well as the text based one you think is best for audit logs. 
Or you can iterate on the existing FQL functionality to get what you want from 
it.

Chronicle has a human and machine readable text format. Switching between the 
two is a flag. You could use the existing full query log plumbing to write one 
of those if you wanted. You can also convert binary logs to human readable logs 
using fqltool. You can also tail the full query log in human readable format 
using fqltool. I think binary vs text is a bit of a red herring.

Some things a new implementation needs to preserve is support for asynchronous 
logging, bounded memory utilization for queueing, and a reasonably efficient 
approach to appending to a file from multiple threads. It should be possible 
for a beefy node to log all the operations a node is executing and process them 
using a single core if configured to use binary format.



> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14275) Cassandra Driver should send identification information to Server

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-14275:
--
Reviewer: Aleksey Yeschenko

> Cassandra Driver should send identification information to Server
> -
>
> Key: CASSANDRA-14275
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14275
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.x
>
>
> Currently there doesn't seem to be any way to readily identify the driver 
> that clients are using to connect to Cassandra. Add the capability of 
> identifying the driver through metadata information much like how HTTP 
> Clients identify themselves through User-Agent HTTP header. This is useful 
> for debugging in large deployments where clients tend to use different 
> drivers, wrappers and language bindings to connect to Cassandra. This can 
> help surface issues as well as help detect clients which are using older or 
> unsupported drivers.
> The identification information should be a string that unambiguously 
> identifies the driver. It should include information such as the name of the 
> driver, it's version, CQL version, Platform (Linux, macOS, Windows, etc.) and 
> architecture (x86, x86_64).
> We should surface this information in `nodetool clientstats` command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14128) Prune dead links from cassandra.apache.org

[Kenneth Brotman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386213#comment-16386213
 ] 

Kenneth Brotman commented on CASSANDRA-14128:
-

Maybe it would be better to split this Jira up into a few smaller ones that can 
each resolve separately.

> Prune dead links from cassandra.apache.org
> --
>
> Key: CASSANDRA-14128
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14128
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: Jeff Jirsa
>Assignee: Kurt Greaves
>Priority: Trivial
>  Labels: lhf
> Attachments: 14128-site.patch
>
>
> Lots of dead links on the site, including the homepage. Should do some 
> pruning cleanup.
> Site repo is [here|https://svn.apache.org/repos/asf/cassandra/site/] (in svn) 
> for anyone who wishes to give this a shot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13686) Fix documentation typo

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386164#comment-16386164
 ] 

Michael Shuler commented on CASSANDRA-13686:


[d6982cd|https://github.com/apache/cassandra/commit/d6982cd221ae6482cbe1cb796fe73d61160a89e0]
 committed to cassandra-3.11 branch and merged to trunk. Thank you for the 
patch [~andrewwuan].

> Fix documentation typo
> --
>
> Key: CASSANDRA-13686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13686
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: An Wu
>Assignee: Michael Shuler
>Priority: Trivial
>  Labels: docuentation
> Fix For: 4.0, 3.11.3
>
> Attachments: fix.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Fix documentation typo under 
> {quote}doc/html/cql/definitions.html#constants{quote}
> and
> {quote}doc/html/cql/ddl.html#the-clustering-columns{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13686) Fix documentation typo

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler resolved CASSANDRA-13686.

Resolution: Fixed

> Fix documentation typo
> --
>
> Key: CASSANDRA-13686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13686
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: An Wu
>Assignee: Michael Shuler
>Priority: Trivial
>  Labels: docuentation
> Fix For: 4.0, 3.11.3
>
> Attachments: fix.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Fix documentation typo under 
> {quote}doc/html/cql/definitions.html#constants{quote}
> and
> {quote}doc/html/cql/ddl.html#the-clustering-columns{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13686) Fix documentation typo

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler reassigned CASSANDRA-13686:
--

Assignee: Michael Shuler

> Fix documentation typo
> --
>
> Key: CASSANDRA-13686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13686
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: An Wu
>Assignee: Michael Shuler
>Priority: Trivial
>  Labels: docuentation
> Fix For: 4.0, 3.11.3
>
> Attachments: fix.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Fix documentation typo under 
> {quote}doc/html/cql/definitions.html#constants{quote}
> and
> {quote}doc/html/cql/ddl.html#the-clustering-columns{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13686) Fix documentation typo

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13686:
---
Fix Version/s: (was: 3.11.x)
   3.11.3
   4.0

> Fix documentation typo
> --
>
> Key: CASSANDRA-13686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13686
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: An Wu
>Assignee: Michael Shuler
>Priority: Trivial
>  Labels: docuentation
> Fix For: 4.0, 3.11.3
>
> Attachments: fix.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Fix documentation typo under 
> {quote}doc/html/cql/definitions.html#constants{quote}
> and
> {quote}doc/html/cql/ddl.html#the-clustering-columns{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/3] cassandra git commit: Fix CQL documentation typos

[mshuler]

Fix CQL documentation typos

Patch by An Wu; Reviewed by Michael Shuler for CASSANDRA-13686


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d6982cd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d6982cd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d6982cd2

Branch: refs/heads/trunk
Commit: d6982cd221ae6482cbe1cb796fe73d61160a89e0
Parents: 465d869
Author: An Wu 
Authored: Mon Mar 5 08:48:37 2018 -0600
Committer: Michael Shuler 
Committed: Mon Mar 5 08:48:37 2018 -0600

--
 doc/source/cql/ddl.rst | 2 +-
 doc/source/cql/definitions.rst | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6982cd2/doc/source/cql/ddl.rst
--
diff --git a/doc/source/cql/ddl.rst b/doc/source/cql/ddl.rst
index be87a7d..3027775 100644
--- a/doc/source/cql/ddl.rst
+++ b/doc/source/cql/ddl.rst
@@ -374,7 +374,7 @@ then the rows (which all belong to the same partition) are 
all stored internally
 ``b`` column (the order they are displayed above). So where the partition key 
of the table allows to group rows on the
 same replica set, the clustering columns controls how those rows are stored on 
the replica. That sorting allows the
 retrieval of a range of rows within a partition (for instance, in the example 
above, ``SELECT * FROM t WHERE a = 0 AND b
-> 1 and b <= 3``) very efficient.
+> 1 and b <= 3``) to be very efficient.
 
 
 .. _create-table-options:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6982cd2/doc/source/cql/definitions.rst
--
diff --git a/doc/source/cql/definitions.rst b/doc/source/cql/definitions.rst
index e54bcd7..d4a5b59 100644
--- a/doc/source/cql/definitions.rst
+++ b/doc/source/cql/definitions.rst
@@ -102,7 +102,7 @@ In other words:
 - A string constant is an arbitrary sequence of characters enclosed by 
single-quote(``'``). A single-quote
   can be included by repeating it, e.g. ``'It''s raining today'``. Those are 
not to be confused with quoted
   :ref:`identifiers` that use double-quotes. Alternatively, a string can be 
defined by enclosing the arbitrary sequence
-  of characters by two dollar characters, in which case single-quote can be 
use without escaping (``$$It's raining
+  of characters by two dollar characters, in which case single-quote can be 
used without escaping (``$$It's raining
   today$$``). That latter form is often used when defining :ref:`user-defined 
functions ` to avoid having to
   escape single-quote characters in function body (as they are more likely to 
occur than ``$$``).
 - Integer, float and boolean constant are defined as expected. Note however 
than float allows the special ``NaN`` and


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[mshuler]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/551744a9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/551744a9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/551744a9

Branch: refs/heads/trunk
Commit: 551744a990022fa7c5ff76529817b05590d89b68
Parents: 2201e36 d6982cd
Author: Michael Shuler 
Authored: Mon Mar 5 08:51:26 2018 -0600
Committer: Michael Shuler 
Committed: Mon Mar 5 08:51:26 2018 -0600

--
 doc/source/cql/ddl.rst | 2 +-
 doc/source/cql/definitions.rst | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/551744a9/doc/source/cql/ddl.rst
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/551744a9/doc/source/cql/definitions.rst
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/3] cassandra git commit: Fix CQL documentation typos

[mshuler]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 465d86904 -> d6982cd22
  refs/heads/trunk 2201e364b -> 551744a99


Fix CQL documentation typos

Patch by An Wu; Reviewed by Michael Shuler for CASSANDRA-13686


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d6982cd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d6982cd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d6982cd2

Branch: refs/heads/cassandra-3.11
Commit: d6982cd221ae6482cbe1cb796fe73d61160a89e0
Parents: 465d869
Author: An Wu 
Authored: Mon Mar 5 08:48:37 2018 -0600
Committer: Michael Shuler 
Committed: Mon Mar 5 08:48:37 2018 -0600

--
 doc/source/cql/ddl.rst | 2 +-
 doc/source/cql/definitions.rst | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6982cd2/doc/source/cql/ddl.rst
--
diff --git a/doc/source/cql/ddl.rst b/doc/source/cql/ddl.rst
index be87a7d..3027775 100644
--- a/doc/source/cql/ddl.rst
+++ b/doc/source/cql/ddl.rst
@@ -374,7 +374,7 @@ then the rows (which all belong to the same partition) are 
all stored internally
 ``b`` column (the order they are displayed above). So where the partition key 
of the table allows to group rows on the
 same replica set, the clustering columns controls how those rows are stored on 
the replica. That sorting allows the
 retrieval of a range of rows within a partition (for instance, in the example 
above, ``SELECT * FROM t WHERE a = 0 AND b
-> 1 and b <= 3``) very efficient.
+> 1 and b <= 3``) to be very efficient.
 
 
 .. _create-table-options:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6982cd2/doc/source/cql/definitions.rst
--
diff --git a/doc/source/cql/definitions.rst b/doc/source/cql/definitions.rst
index e54bcd7..d4a5b59 100644
--- a/doc/source/cql/definitions.rst
+++ b/doc/source/cql/definitions.rst
@@ -102,7 +102,7 @@ In other words:
 - A string constant is an arbitrary sequence of characters enclosed by 
single-quote(``'``). A single-quote
   can be included by repeating it, e.g. ``'It''s raining today'``. Those are 
not to be confused with quoted
   :ref:`identifiers` that use double-quotes. Alternatively, a string can be 
defined by enclosing the arbitrary sequence
-  of characters by two dollar characters, in which case single-quote can be 
use without escaping (``$$It's raining
+  of characters by two dollar characters, in which case single-quote can be 
used without escaping (``$$It's raining
   today$$``). That latter form is often used when defining :ref:`user-defined 
functions ` to avoid having to
   escape single-quote characters in function body (as they are more likely to 
occur than ``$$``).
 - Integer, float and boolean constant are defined as expected. Note however 
than float allows the special ``NaN`` and


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386137#comment-16386137
 ] 

Stefan Podkowinski commented on CASSANDRA-12151:


[~eanujwa] wrote:
{quote}If auditing for a database application user is only at application 
level, what would happen in case direct queries are executed on the database 
using the the application user credentials (using tools such as cqlsh)?
{quote}
If the operator decided not to enable audit logging for the application users, 
nothing would get logged. If someone steals the credentials, none of the 
activities would show up in the logs. If you think that this is unacceptable, 
then you can enable auditing for the app user as well, which I would not, but 
it’s really up to you.

[~jolynch] wrote:
{quote}Doesn't the category filter adequately achieve this (you could exclude 
DML or QUERY)? Do we need per user query logging when there is already per user 
permissions limiting their access to the database in the first place?
{quote}
Although I understand the motivation behind the query categories from a 
developer perspective, it’s probably a feature that is not as easy to 
understand and handled by operators. Let alone managers and compliance 
officers, who lack the technical understanding for discussing which categories 
needs to be enabled for which tables. Once you’ll notice that you have to limit 
the audit logging due to the performance impact, it’s going to be a rather 
painful discussion how categories and tables should be configured in this 
regard. I’d rather give ops people more simpler options that are easy to 
understand and reason about, when discussed between ops and non-technical 
stakeholders. Offering “all or nothing” audit logging on user basis would fit 
this requirements for me (at least for CQL, login events could be a different 
story). 

I'd also really like to avoid luring people into the idea that Cassandra would 
be a one-stop solution for audit logging and you don't have to implement your 
own logging on application level anymore, since Cassandra does handle this for 
you. It probably won't, as logging all statements for applications will be too 
much of a performance impact. Now the question for me is to offer operators 
options to optimize their way out of this situation, or simply tell people that 
it's not possible to handle this use-case (audit logging for application users 
on high-load systems) in Cassandra. I'd clearly go with the later, as many 
users think that Cassandra already puts too much of a burden on operators.

 
{quote}While I agree use case #1 (security) does not require this, use cases #2 
and #3 very much do. For #2 or similar you typically have to prove that only 
authorized applications manipulated the database and a typical way to do that 
is to produce query logs showing that only trusted application IP addresses and 
specific credentials made DML statements (but QUERY is less important). For #3 
the requirements are even greater, e.g. you may have to be able to prove that 
user data was not exfiltrated at all, requiring auditing of QUERY statements. 
Yes it's higher overhead but if you can turn it off with the category filters I 
think it's fine don't you?
{quote}
Can you really “prove” unauthorised data manipulation did *not* happen through 
the audit logs? Say I have an application that is updating a credit score and 
now I use that user to do an update manually, how would you ever notice that 
from the audit logs? My manual statement looks just like the million others 
executed by the app. IP based access restrictions should be in place anyways, 
that’s also not something that would necessarily give you additional clues.

As was already suggested, let’s take an incremental approach that will allow us 
to introduce a simple audit logging that can be understood by both technical 
and non-technical people and configured in little time. If any external 
compliance authority raises some concerns, e.g. that DCL statements need to be 
handled differently from queries, we can take care of that. But any 90% 
solutions would still be a big win over what we currently have.

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log 

[jira] [Commented] (CASSANDRA-14170) Loss of digits when doing CAST from varint/bigint to decimal

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386024#comment-16386024
 ] 

Andrés de la Peña commented on CASSANDRA-14170:
---

Overall the patch looks good to me. However, the new test 
[{{testNoLossOfPrecisionForCastToDecimal}}|https://github.com/blerer/cassandra/blob/4b1f5804837ae3164bb183fcb62077683621651a/test/unit/org/apache/cassandra/cql3/functions/CastFctsTest.java#L205-L212]
 misses the {{@Test}} annotation. Indeed, it fails due to [a column names 
mismatch at the insert 
statement|https://github.com/blerer/cassandra/blob/4b1f5804837ae3164bb183fcb62077683621651a/test/unit/org/apache/cassandra/cql3/functions/CastFctsTest.java#L208].
 I think it should be [this 
way|https://github.com/adelapena/cassandra/commit/be44415a3f61e13720889cfd44482ee1142e8c1d].
 If it's ok for you it can be addressed during commit.

Nitpick: there is a misssed blank line in {{getDecimalConversionFunction}} 
JavaDoc, 
[here|https://github.com/adelapena/cassandra/commit/86f8121f95adb60eef85edbcaf9fb3917ef5207f].

> Loss of digits when doing CAST from varint/bigint to decimal
> 
>
> Key: CASSANDRA-14170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14170
> Project: Cassandra
>  Issue Type: Bug
> Environment: Tested with Cassandra 3.11.1 but this issue is present 
> since the implementation of cast functions.
>Reporter: Daniel Fiala
>Assignee: Benjamin Lerer
>Priority: Minor
>
> Cast functions from numeric types to decimal type are implemented as 
> conversion to double first and then from double to decimal: 
> [https://github.com/apache/cassandra/compare/trunk...blerer:10310-3.0#diff-6aa4a8f76df6c30c5bb4026b8c9251eeR80].
> This can cause loss of digits for big values stored in varint or bigint. It 
> is probably unexpected because decimal can store such values precisely.
> Examples:
> {{cqlsh> CREATE TABLE cast_bigint_test(k int PRIMARY KEY, bigint_clmn 
> bigint);}}
>  {{cqlsh> INSERT INTO cast_bigint_test(k, decimal_clmn) VALUES(2, 
> 9223372036854775807);}}
>  {{cqlsh> SELECT CAST(bigint_clmn AS decimal) FROM cast_bigint_test;}}
>  {{cast(bigint_clmn as decimal)}}
>  {{--}}
>  {{9.223372036854776E+18}}
>  {{(1 rows)}}
> {{cqlsh> CREATE TABLE cast_varint_test (k int PRIMARY KEY, varint_clmn 
> varint);}}
>  {{cqlsh> INSERT INTO cast_varint_test(k, varint_clmn) values(2, 
> 1234567890123456789);}}
>  {{cqlsh> SELECT CAST(varint_clmn AS decimal) FROM cast_varint_test;}}
>  {{cast(varint_clmn as decimal)}}
>  {{--}}
> 1.23456789012345677E+18
>  {{(1 rows)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14284) Chunk checksum test needs to occur before uncompress to avoid JVM crash

[Gil Tene (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gil Tene updated CASSANDRA-14284:
-
Description: 
While checksums are (generally) performed on compressed data, the checksum test 
when reading is currently (in all variants of C* 2.x, 3.x I've looked at) done 
[on the compressed data] only after the uncompress operation has completed. 

The issue here is that LZ4_decompress_fast (as documented in e.g. 
[https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L214)] can result in memory 
overruns when provided with malformed source data. This in turn can (and does, 
e.g. in CASSANDRA-13757) lead to JVM crashes during the uncompress of corrupted 
chunks. The checksum operation would obviously detect the issue, but we'd never 
get to it if the JVM crashes first.

Moving the checksum test of the compressed data to before the uncompress 
operation (in cases where the checksum is done on compressed data) will resolve 
this issue.

-

The check-only-after-doing-the-decompress logic appears to be in all current 
releases.

Here are some samples at different evolution points :

3.11.2:

[https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L146]

https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L207

 

3.5:

 
[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]

[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]

2.1.17:

 
[https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]

  was:
While checksums are (generally) performed on compressed data, the checksum test 
when reading is currently (in all variants of C* 2.x, 3.x I've looked at) done 
[on the compressed data] only after the uncompress operation has completed. 

The issue here is that LZ4_decompress_fast (as documented in e.g. 
[https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L214)] can result in memory 
overruns when provided with malformed source data. This in turn can (and does, 
e.g. in CASSANDRA-13757) lead to JVM crashes during the uncompress of corrupted 
chunks. The checksum operation would obviously detect the issue, but we'd never 
get to it if the JVM crashes first.

Moving the checksum test of the compressed data to before the uncompress 
operation (in cases where the checksum is done on compressed data) will resolve 
this issue.

-

The check-only-after-doing-the-decompress logic appears to be in all current 
releases.

Here are some samples at different evolution points :

3.11.2:

[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]

[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L198]

3.5:

 
[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]

[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]

2.1.17:

 
[https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]


> Chunk checksum test needs to occur before uncompress to avoid JVM crash
> ---
>
> Key: CASSANDRA-14284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14284
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: The check-only-after-doing-the-decompress logic appears 
> to be in all current releases.
> Here are some samples at different evolution points :
> 3.11.2:
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L198
> 3.5:
>  
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196
> 2.1.17:
>  
> [https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]
>  
>Reporter: Gil Tene
>Assignee: Benjamin 

[jira] [Updated] (CASSANDRA-14284) Chunk checksum test needs to occur before uncompress to avoid JVM crash

[Gil Tene (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gil Tene updated CASSANDRA-14284:
-
Environment: 
The check-only-after-doing-the-decompress logic appears to be in all current 
releases.

Here are some samples at different evolution points :

3.11.2:

[https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L146]

https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L207

 

3.5:

 
[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]

[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]

2.1.17:

 
[https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]

 

  was:
The check-only-after-doing-the-decompress logic appears to be in all current 
releases.

Here are some samples at different evolution points :

3.11.2:

https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135

https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L198

3.5:

 
[https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]

https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196

2.1.17:

 
[https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]

 


> Chunk checksum test needs to occur before uncompress to avoid JVM crash
> ---
>
> Key: CASSANDRA-14284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14284
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: The check-only-after-doing-the-decompress logic appears 
> to be in all current releases.
> Here are some samples at different evolution points :
> 3.11.2:
> [https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L146]
> https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L207
>  
> 3.5:
>  
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]
> 2.1.17:
>  
> [https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]
>  
>Reporter: Gil Tene
>Assignee: Benjamin Lerer
>Priority: Major
>
> While checksums are (generally) performed on compressed data, the checksum 
> test when reading is currently (in all variants of C* 2.x, 3.x I've looked 
> at) done [on the compressed data] only after the uncompress operation has 
> completed. 
> The issue here is that LZ4_decompress_fast (as documented in e.g. 
> [https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L214)] can result in memory 
> overruns when provided with malformed source data. This in turn can (and 
> does, e.g. in CASSANDRA-13757) lead to JVM crashes during the uncompress of 
> corrupted chunks. The checksum operation would obviously detect the issue, 
> but we'd never get to it if the JVM crashes first.
> Moving the checksum test of the compressed data to before the uncompress 
> operation (in cases where the checksum is done on compressed data) will 
> resolve this issue.
> -
> The check-only-after-doing-the-decompress logic appears to be in all current 
> releases.
> Here are some samples at different evolution points :
> 3.11.2:
> [https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L146]
> https://github.com/apache/cassandra/blob/cassandra-3.11.2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java#L207
>  
> 3.5:
>  
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]
> 2.1.17:
>  
> 

[jira] [Commented] (CASSANDRA-14244) Some tests in read_repair_test are flakey

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385873#comment-16385873
 ] 

Marcus Eriksson commented on CASSANDRA-14244:
-

+1

> Some tests in read_repair_test are flakey
> -
>
> Key: CASSANDRA-14244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14244
> Project: Cassandra
>  Issue Type: Test
>  Components: Testing
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
>  Labels: dtest
>
> Since being refactored for CASSANDRA-14134, 
> {{test_alter_rf_and_run_read_repair}} and {{test_read_repair_chance}} in 
> {{read_repair_test.TestReadRepair}} are flakey and regularly fail on all 
> branches. The problem is that the inital setup of these two tests doesn't 
> explicitly set the {{read_repair_chance}} or {{dclocal_read_repair_chance}} 
> properties on the test table. As a consequence, read repairs are sometimes 
> probabilistically triggered and query results don't match the expectations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14244) Some tests in read_repair_test are flakey

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14244:

Status: Ready to Commit  (was: Patch Available)

> Some tests in read_repair_test are flakey
> -
>
> Key: CASSANDRA-14244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14244
> Project: Cassandra
>  Issue Type: Test
>  Components: Testing
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
>  Labels: dtest
>
> Since being refactored for CASSANDRA-14134, 
> {{test_alter_rf_and_run_read_repair}} and {{test_read_repair_chance}} in 
> {{read_repair_test.TestReadRepair}} are flakey and regularly fail on all 
> branches. The problem is that the inital setup of these two tests doesn't 
> explicitly set the {{read_repair_chance}} or {{dclocal_read_repair_chance}} 
> properties on the test table. As a consequence, read repairs are sometimes 
> probabilistically triggered and query results don't match the expectations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14286) IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-14286:
--

Assignee: Benjamin Lerer

> IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY
> 
>
> Key: CASSANDRA-14286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14286
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Kubernetes cluster using cassandra:3.11.1 Docker image.
>Reporter: Szymon Acedański
>Assignee: Benjamin Lerer
>Priority: Major
> Fix For: 3.11.x
>
> Attachments: orderbug-traceback.txt
>
>
> When running the following code:
> {code}
> public class CassandraJsonOrderingBug {
> public static void main(String[] args) {
> Session session = CassandraFactory.getSession();
> session.execute("CREATE TABLE thebug ( PRIMARY KEY (a, b), a INT, b 
> INT)");
> try {
> session.execute("INSERT INTO thebug (a, b) VALUES (20, 30)");
> session.execute("INSERT INTO thebug (a, b) VALUES (100, 200)");
> Statement statement = new SimpleStatement("SELECT JSON a, b FROM 
> thebug WHERE a IN (20, 100) ORDER BY b");
> statement.setFetchSize(Integer.MAX_VALUE);
> for (Row w: session.execute(statement)) {
> System.out.println(w.toString());
> }
> } finally {
> session.execute("DROP TABLE thebug");
> }
> }
> }
> {code}
> The following exception is thrown server-side:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>   at java.util.Collections$SingletonList.get(Collections.java:4815) 
> ~[na:1.8.0_151]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1297)
>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1284)
>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>   at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) 
> ~[na:1.8.0_151]
>   at java.util.TimSort.sort(TimSort.java:220) ~[na:1.8.0_151]
>   at java.util.Arrays.sort(Arrays.java:1512) ~[na:1.8.0_151]
>   at java.util.ArrayList.sort(ArrayList.java:1460) ~[na:1.8.0_151]
>   at java.util.Collections.sort(Collections.java:175) ~[na:1.8.0_151]
> {noformat}
> (full traceback attached)
> The accessed index is the index of the sorted column in the SELECT JSON 
> fields list.
> Similarly, if the select clause is changed to
> SELECT JSON b, a FROM thebug WHERE a IN (20, 100) ORDER BY b
> then the query finishes, but the output is sorted incorrectly (by textual 
> JSON representation):
> {noformat}
> Row[{"b": 200, "a": 100}]
> Row[{"b": 30, "a": 20}]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14284) Chunk checksum test needs to occur before uncompress to avoid JVM crash

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-14284:
--

Assignee: Benjamin Lerer

> Chunk checksum test needs to occur before uncompress to avoid JVM crash
> ---
>
> Key: CASSANDRA-14284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14284
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: The check-only-after-doing-the-decompress logic appears 
> to be in all current releases.
> Here are some samples at different evolution points :
> 3.11.2:
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L198
> 3.5:
>  
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196
> 2.1.17:
>  
> [https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]
>  
>Reporter: Gil Tene
>Assignee: Benjamin Lerer
>Priority: Major
>
> While checksums are (generally) performed on compressed data, the checksum 
> test when reading is currently (in all variants of C* 2.x, 3.x I've looked 
> at) done [on the compressed data] only after the uncompress operation has 
> completed. 
> The issue here is that LZ4_decompress_fast (as documented in e.g. 
> [https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L214)] can result in memory 
> overruns when provided with malformed source data. This in turn can (and 
> does, e.g. in CASSANDRA-13757) lead to JVM crashes during the uncompress of 
> corrupted chunks. The checksum operation would obviously detect the issue, 
> but we'd never get to it if the JVM crashes first.
> Moving the checksum test of the compressed data to before the uncompress 
> operation (in cases where the checksum is done on compressed data) will 
> resolve this issue.
> -
> The check-only-after-doing-the-decompress logic appears to be in all current 
> releases.
> Here are some samples at different evolution points :
> 3.11.2:
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L198]
> 3.5:
>  
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L135]
> [https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L196]
> 2.1.17:
>  
> [https://github.com/apache/cassandra/blob/cassandra-2.1.17/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java#L122]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12813) NPE in auth for bootstrapping node

[Avraham Kalvo (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385770#comment-16385770
 ] 

Avraham Kalvo commented on CASSANDRA-12813:
---

Thank you Jeff.




> NPE in auth for bootstrapping node
> --
>
> Key: CASSANDRA-12813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12813
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Charles Mims
>Assignee: Alex Petrov
>Priority: Major
> Fix For: 2.2.9, 3.0.10, 3.10
>
>
> {code}
> ERROR [SharedPool-Worker-1] 2016-10-19 21:40:25,991 Message.java:617 - 
> Unexpected exception during request; channel = [id: 0x15eb017f, / omitted>:40869 => /10.0.0.254:9042]
> java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:144)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:86)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:54)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:182)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  [apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  [apache-cassandra-3.0.9.jar:3.0.9]
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_101]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  [apache-cassandra-3.0.9.jar:3.0.9]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [apache-cassandra-3.0.9.jar:3.0.9]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}
> I have a node that has been joining for around 24 hours.  My application is 
> configured with the IP address of the joining node in the list of nodes to 
> connect to (ruby driver), and I have been getting around 200 events of this 
> NPE per hour.  I removed the IP of the joining node from the list of nodes 
> for my app to connect to and the errors stopped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14210) Optimize SSTables upgrade task scheduling

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14210:

   Resolution: Fixed
Fix Version/s: (was: 4.x)
   3.11.3
   3.0.17
   4.0
Reproduced In: 3.0.15, 2.2.11  (was: 2.2.11, 3.0.15)
   Status: Resolved  (was: Ready to Commit)

committed as {{f88ec9357de406daad0f795951f17e5f854ade10}} - thanks!

> Optimize SSTables upgrade task scheduling
> -
>
> Key: CASSANDRA-14210
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14210
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Oleksandr Shulgin
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 4.0, 3.0.17, 3.11.3
>
>
> When starting the SSTable-rewrite process by running {{nodetool 
> upgradesstables --jobs N}}, with N > 1, not all of the provided N slots are 
> used.
> For example, we were testing with {{concurrent_compactors=5}} and {{N=4}}.  
> What we observed both for version 2.2 and 3.0, is that initially all 4 
> provided slots are used for "Upgrade sstables" compactions, but later when 
> some of the 4 tasks are finished, no new tasks are scheduled immediately.  It 
> takes the last of the 4 tasks to finish before new 4 tasks would be 
> scheduled.  This happens on every node we've observed.
> This doesn't utilize available resources to the full extent allowed by the 
> --jobs N parameter.  In the field, on a cluster of 12 nodes with 4-5 TiB data 
> each, we've seen that the whole process was taking more than 7 days, instead 
> of estimated 1.5-2 days (provided there would be close to full N slots 
> utilization).
> Instead, new tasks should be scheduled as soon as there is a free compaction 
> slot.
> Additionally, starting from the biggest SSTables could further reduce the 
> total time required for the whole process to finish on any given node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/465d8690
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/465d8690
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/465d8690

Branch: refs/heads/trunk
Commit: 465d86904908b4d06cb4a3172dd5c6421ddef1b8
Parents: c494696 f88ec93
Author: Marcus Eriksson 
Authored: Mon Mar 5 09:15:15 2018 +0100
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:15:15 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../org/apache/cassandra/utils/FBUtilities.java | 36 +++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 37 
 4 files changed, 80 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/CHANGES.txt
--
diff --cc CHANGES.txt
index 33d5da3,5599906..163d787
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,43 -1,18 +1,44 @@@
 -3.0.17
 +3.11.3
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Fully utilise specified compaction threads (CASSANDRA-14210)
   * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 -Merged from 2.1:
 - * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  Merged from 2.2:
   * Backport circleci yaml (CASSANDRA-14240)
 +Merged from 2.1:
 + * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  
 -3.0.16
 - * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)
 - * Add MinGW uname check to start scripts (CASSANDRA-12940)
 - * Protect against overflow of local expiration time (CASSANDRA-14092)
 +
 +3.11.2
 + * Fix ReadCommandTest (CASSANDRA-14234)
 + * Remove trailing period from latency reports at keyspace level 
(CASSANDRA-14233)
 + * Backport CASSANDRA-13080: Use new token allocation for non bootstrap case 
as well (CASSANDRA-14212)
 + * Remove dependencies on JVM internal classes from JMXServerUtils 
(CASSANDRA-14173) 
 + * Add DEFAULT, UNSET, MBEAN and MBEANS to `ReservedKeywords` 
(CASSANDRA-14205)
 + * Add Unittest for schema migration fix (CASSANDRA-14140)
 + * Print correct snitch info from nodetool describecluster (CASSANDRA-13528)
 + * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
 + * Enable CDC unittest (CASSANDRA-14141)
 + * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
 + * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 + * Correctly count range tombstones in traces and tombstone thresholds 
(CASSANDRA-8527)
 +Merged from 3.0:
 + * Add MinGW uname check to start scripts (CASSANDRA-12840)
   * Use the correct digest file and reload sstable metadata in nodetool verify 
(CASSANDRA-14217)
   * Handle failure when mutating repaired status in Verifier (CASSANDRA-13933)
 - * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
   * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/src/java/org/apache/cassandra/utils/FBUtilities.java
--

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[marcuse]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2201e364
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2201e364
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2201e364

Branch: refs/heads/trunk
Commit: 2201e364b5b1085c18098ba10f48a4514f9c5dfc
Parents: 68b8137 465d869
Author: Marcus Eriksson 
Authored: Mon Mar 5 09:15:24 2018 +0100
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:15:24 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../org/apache/cassandra/utils/FBUtilities.java | 36 +++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 37 
 4 files changed, 80 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2201e364/CHANGES.txt
--
diff --cc CHANGES.txt
index 9d0b483,163d787..72554c2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,209 -1,13 +1,210 @@@
 +4.0
 + * Use Murmur3 for validation compactions (CASSANDRA-14002)
 + * Comma at the end of the seed list is interpretated as localhost 
(CASSANDRA-14285)
 + * Refactor read executor and response resolver, abstract read repair 
(CASSANDRA-14058)
 + * Add optional startup delay to wait until peers are ready (CASSANDRA-13993)
 + * Add a few options to nodetool verify (CASSANDRA-14201)
 + * CVE-2017-5929 Security vulnerability and redefine default log rotation 
policy (CASSANDRA-14183)
 + * Use JVM default SSL validation algorithm instead of custom default 
(CASSANDRA-13259)
 + * Better document in code InetAddressAndPort usage post 7544, incorporate 
port into UUIDGen node (CASSANDRA-14226)
 + * Fix sstablemetadata date string for minLocalDeletionTime (CASSANDRA-14132)
 + * Make it possible to change neverPurgeTombstones during runtime 
(CASSANDRA-14214)
 + * Remove GossipDigestSynVerbHandler#doSort() (CASSANDRA-14174)
 + * Add nodetool clientlist (CASSANDRA-13665)
 + * Revert ProtocolVersion changes from CASSANDRA-7544 (CASSANDRA-14211)
 + * Non-disruptive seed node list reload (CASSANDRA-14190)
 + * Nodetool tablehistograms to print statics for all the tables 
(CASSANDRA-14185)
 + * Migrate dtests to use pytest and python3 (CASSANDRA-14134)
 + * Allow storage port to be configurable per node (CASSANDRA-7544)
 + * Make sub-range selection for non-frozen collections return null instead of 
empty (CASSANDRA-14182)
 + * BloomFilter serialization format should not change byte ordering 
(CASSANDRA-9067)
 + * Remove unused on-heap BloomFilter implementation (CASSANDRA-14152)
 + * Delete temp test files on exit (CASSANDRA-14153)
 + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867)
 + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066)
 + * Fix cassandra-stress startup failure (CASSANDRA-14106)
 + * Remove initialDirectories from CFS (CASSANDRA-13928)
 + * Fix trivial log format error (CASSANDRA-14015)
 + * Allow sstabledump to do a json object per partition (CASSANDRA-13848)
 + * Add option to optimise merkle tree comparison across replicas 
(CASSANDRA-3200)
 + * Remove unused and deprecated methods from AbstractCompactionStrategy 
(CASSANDRA-14081)
 + * Fix Distribution.average in cassandra-stress (CASSANDRA-14090)
 + * Support a means of logging all queries as they were invoked 
(CASSANDRA-13983)
 + * Presize collections (CASSANDRA-13760)
 + * Add GroupCommitLogService (CASSANDRA-13530)
 + * Parallelize initial materialized view build (CASSANDRA-12245)
 + * Fix flaky SecondaryIndexManagerTest.assert[Not]MarkedAsBuilt 
(CASSANDRA-13965)
 + * Make LWTs send resultset metadata on every request (CASSANDRA-13992)
 + * Fix flaky indexWithFailedInitializationIsNotQueryableAfterPartialRebuild 
(CASSANDRA-13963)
 + * Introduce leaf-only iterator (CASSANDRA-9988)
 + * Upgrade Guava to 23.3 and Airline to 0.8 (CASSANDRA-13997)
 + * Allow only one concurrent call to StatusLogger (CASSANDRA-12182)
 + * Refactoring to specialised functional interfaces (CASSANDRA-13982)
 + * Speculative retry should allow more friendly params (CASSANDRA-13876)
 + * Throw exception if we send/receive repair messages to incompatible nodes 
(CASSANDRA-13944)
 + * Replace usages of MessageDigest with Guava's Hasher (CASSANDRA-13291)
 + * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728)
 + * Fix some alerts raised by static analysis (CASSANDRA-13799)
 + * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593)
 + * Add result set metadata to prepared statement MD5 hash calculation 
(CASSANDRA-10786)
 + * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941)
 + * Expose recent histograms in 

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/465d8690
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/465d8690
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/465d8690

Branch: refs/heads/cassandra-3.11
Commit: 465d86904908b4d06cb4a3172dd5c6421ddef1b8
Parents: c494696 f88ec93
Author: Marcus Eriksson 
Authored: Mon Mar 5 09:15:15 2018 +0100
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:15:15 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../org/apache/cassandra/utils/FBUtilities.java | 36 +++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 37 
 4 files changed, 80 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/CHANGES.txt
--
diff --cc CHANGES.txt
index 33d5da3,5599906..163d787
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,43 -1,18 +1,44 @@@
 -3.0.17
 +3.11.3
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Fully utilise specified compaction threads (CASSANDRA-14210)
   * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 -Merged from 2.1:
 - * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  Merged from 2.2:
   * Backport circleci yaml (CASSANDRA-14240)
 +Merged from 2.1:
 + * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
  
 -3.0.16
 - * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)
 - * Add MinGW uname check to start scripts (CASSANDRA-12940)
 - * Protect against overflow of local expiration time (CASSANDRA-14092)
 +
 +3.11.2
 + * Fix ReadCommandTest (CASSANDRA-14234)
 + * Remove trailing period from latency reports at keyspace level 
(CASSANDRA-14233)
 + * Backport CASSANDRA-13080: Use new token allocation for non bootstrap case 
as well (CASSANDRA-14212)
 + * Remove dependencies on JVM internal classes from JMXServerUtils 
(CASSANDRA-14173) 
 + * Add DEFAULT, UNSET, MBEAN and MBEANS to `ReservedKeywords` 
(CASSANDRA-14205)
 + * Add Unittest for schema migration fix (CASSANDRA-14140)
 + * Print correct snitch info from nodetool describecluster (CASSANDRA-13528)
 + * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
 + * Enable CDC unittest (CASSANDRA-14141)
 + * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
 + * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 + * Correctly count range tombstones in traces and tombstone thresholds 
(CASSANDRA-8527)
 +Merged from 3.0:
 + * Add MinGW uname check to start scripts (CASSANDRA-12840)
   * Use the correct digest file and reload sstable metadata in nodetool verify 
(CASSANDRA-14217)
   * Handle failure when mutating repaired status in Verifier (CASSANDRA-13933)
 - * Close socket on error during connect on OutboundTcpConnection 
(CASSANDRA-9630)
   * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/465d8690/src/java/org/apache/cassandra/utils/FBUtilities.java
--

[2/6] cassandra git commit: Fully utilise specified compaction threads (jobs)

[marcuse]

Fully utilise specified compaction threads (jobs)

Patch by Kurt Greaves; reviewed by marcuse for CASSANDRA-14210


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f88ec935
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f88ec935
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f88ec935

Branch: refs/heads/cassandra-3.11
Commit: f88ec9357de406daad0f795951f17e5f854ade10
Parents: 79cead0
Author: kurt 
Authored: Mon Feb 12 21:06:34 2018 +
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:09:52 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../io/sstable/format/SSTableReader.java|  2 +
 .../org/apache/cassandra/utils/FBUtilities.java | 36 ++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 39 
 5 files changed, 84 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8cf665e..5599906 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:
  * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 1d54667..a8e6931 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -327,8 +327,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 if (jobs > 0 && futures.size() == jobs)
 {
-FBUtilities.waitOnFutures(futures);
-futures.clear();
+Future f = FBUtilities.waitOnFirstFuture(futures);
+futures.remove(f);
 }
 }
 FBUtilities.waitOnFutures(futures);
@@ -416,8 +416,9 @@ public class CompactionManager implements 
CompactionManagerMBean
 @Override
 public Iterable filterSSTables(LifecycleTransaction 
transaction)
 {
-Iterable sstables = new 
ArrayList<>(transaction.originals());
-Iterator iter = sstables.iterator();
+List sortedSSTables = 
Lists.newArrayList(transaction.originals());
+Collections.sort(sortedSSTables, 
SSTableReader.sizeComparator.reversed());
+Iterator iter = sortedSSTables.iterator();
 while (iter.hasNext())
 {
 SSTableReader sstable = iter.next();
@@ -427,7 +428,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 iter.remove();
 }
 }
-return sstables;
+return sortedSSTables;
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 7e1bc1a..2c94e45 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -152,6 +152,8 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCounted sstableOrdering = 
Ordering.from(sstableComparator);
 
+public static final Comparator sizeComparator = (o1, o2) -> 
Longs.compare(o1.onDiskLength(), o2.onDiskLength());
+
 /**
  * maxDataAge is a timestamp in local server time (e.g. 
System.currentTimeMilli) which represents an upper bound
  * to the newest piece of data stored in the sstable. In other words, this 
sstable does not contain items created

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 

[1/6] cassandra git commit: Fully utilise specified compaction threads (jobs)

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 79cead093 -> f88ec9357
  refs/heads/cassandra-3.11 c4946960a -> 465d86904
  refs/heads/trunk 68b81372c -> 2201e364b


Fully utilise specified compaction threads (jobs)

Patch by Kurt Greaves; reviewed by marcuse for CASSANDRA-14210


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f88ec935
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f88ec935
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f88ec935

Branch: refs/heads/cassandra-3.0
Commit: f88ec9357de406daad0f795951f17e5f854ade10
Parents: 79cead0
Author: kurt 
Authored: Mon Feb 12 21:06:34 2018 +
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:09:52 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../io/sstable/format/SSTableReader.java|  2 +
 .../org/apache/cassandra/utils/FBUtilities.java | 36 ++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 39 
 5 files changed, 84 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8cf665e..5599906 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:
  * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 1d54667..a8e6931 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -327,8 +327,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 if (jobs > 0 && futures.size() == jobs)
 {
-FBUtilities.waitOnFutures(futures);
-futures.clear();
+Future f = FBUtilities.waitOnFirstFuture(futures);
+futures.remove(f);
 }
 }
 FBUtilities.waitOnFutures(futures);
@@ -416,8 +416,9 @@ public class CompactionManager implements 
CompactionManagerMBean
 @Override
 public Iterable filterSSTables(LifecycleTransaction 
transaction)
 {
-Iterable sstables = new 
ArrayList<>(transaction.originals());
-Iterator iter = sstables.iterator();
+List sortedSSTables = 
Lists.newArrayList(transaction.originals());
+Collections.sort(sortedSSTables, 
SSTableReader.sizeComparator.reversed());
+Iterator iter = sortedSSTables.iterator();
 while (iter.hasNext())
 {
 SSTableReader sstable = iter.next();
@@ -427,7 +428,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 iter.remove();
 }
 }
-return sstables;
+return sortedSSTables;
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 7e1bc1a..2c94e45 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -152,6 +152,8 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCounted sstableOrdering = 
Ordering.from(sstableComparator);
 
+public static final Comparator sizeComparator = (o1, o2) -> 
Longs.compare(o1.onDiskLength(), o2.onDiskLength());
+
 /**
  * maxDataAge is a timestamp in local server time (e.g. 
System.currentTimeMilli) which represents an upper bound
  * to the newest piece of data stored in the sstable. In other words, this 
sstable does not contain items created

[3/6] cassandra git commit: Fully utilise specified compaction threads (jobs)

[marcuse]

Fully utilise specified compaction threads (jobs)

Patch by Kurt Greaves; reviewed by marcuse for CASSANDRA-14210


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f88ec935
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f88ec935
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f88ec935

Branch: refs/heads/trunk
Commit: f88ec9357de406daad0f795951f17e5f854ade10
Parents: 79cead0
Author: kurt 
Authored: Mon Feb 12 21:06:34 2018 +
Committer: Marcus Eriksson 
Committed: Mon Mar 5 09:09:52 2018 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 11 +++---
 .../io/sstable/format/SSTableReader.java|  2 +
 .../org/apache/cassandra/utils/FBUtilities.java | 36 ++
 .../apache/cassandra/utils/FBUtilitiesTest.java | 39 
 5 files changed, 84 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8cf665e..5599906 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
 Merged from 2.1:
  * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 1d54667..a8e6931 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -327,8 +327,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 if (jobs > 0 && futures.size() == jobs)
 {
-FBUtilities.waitOnFutures(futures);
-futures.clear();
+Future f = FBUtilities.waitOnFirstFuture(futures);
+futures.remove(f);
 }
 }
 FBUtilities.waitOnFutures(futures);
@@ -416,8 +416,9 @@ public class CompactionManager implements 
CompactionManagerMBean
 @Override
 public Iterable filterSSTables(LifecycleTransaction 
transaction)
 {
-Iterable sstables = new 
ArrayList<>(transaction.originals());
-Iterator iter = sstables.iterator();
+List sortedSSTables = 
Lists.newArrayList(transaction.originals());
+Collections.sort(sortedSSTables, 
SSTableReader.sizeComparator.reversed());
+Iterator iter = sortedSSTables.iterator();
 while (iter.hasNext())
 {
 SSTableReader sstable = iter.next();
@@ -427,7 +428,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 iter.remove();
 }
 }
-return sstables;
+return sortedSSTables;
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 7e1bc1a..2c94e45 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -152,6 +152,8 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCounted sstableOrdering = 
Ordering.from(sstableComparator);
 
+public static final Comparator sizeComparator = (o1, o2) -> 
Longs.compare(o1.onDiskLength(), o2.onDiskLength());
+
 /**
  * maxDataAge is a timestamp in local server time (e.g. 
System.currentTimeMilli) which represents an upper bound
  * to the newest piece of data stored in the sstable. In other words, this 
sstable does not contain items created

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f88ec935/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java