[jira] [Created] (CASSANDRA-13050) ReadCommand.CheckForAbort not monitoring CQL rows for range queries

[Stefania (JIRA)]

Stefania created CASSANDRA-13050:


 Summary: ReadCommand.CheckForAbort not monitoring CQL rows for 
range queries
 Key: CASSANDRA-13050
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13050
 Project: Cassandra
  Issue Type: Bug
  Components: Local Write-Read Paths
Reporter: Stefania
Assignee: Stefania
 Fix For: 3.x


If I understood the iterator transformations introduced by CASSANDRA-9975 
correctly, 
[ReadCommand.CheckForAbort|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/ReadCommand.java#L541]
 should apply itself before returning a partition. At the moment it is applied 
to row iterators for single command partitions, but for range queries I think 
it only monitors the query progress when a new partition is iterated, not when 
a new row is iterated. So for large partitions, we may fail to log a query as 
slow, or fail to abort it if the RPC timeout is exceeded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-13037) DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753581#comment-15753581
 ] 

Stefania edited comment on CASSANDRA-13037 at 12/16/16 6:22 AM:


Thanks for checking. I agree that the threads are not deadlocked, my language 
was imprecise. My best guess is that the non-periodic-tasks thread has not been 
signaled, or it has failed to get the correct segment offset before entering 
awaitUninterruptibly(). I'm multiplexing again 
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex/59/], this time 
with a longer timeout (9.5 minutes rather than 5) and with some error log 
messages that are printed out if it takes longer than 1 minute to receive a 
signal in awaitDiskSync(), code is 
[here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643].
 Normally the entire test succeedes in just over 2 minutes.


was (Author: stefania):
Thanks for checking. I agree that the threads are not deadlocked, my language 
was imprecise. My best guess is that the non-periodic-tasks thread has not been 
signaled, or it has failed to get the correct segment offset before entering 
awaitUninterruptibly(). I'm multiplexing again 
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex/58/], this time 
with a longer timeout (9.5 minutes rather than 5) and with some error log 
messages that are printed out if it takes longer than 1 minute to receive a 
signal in awaitDiskSync(), code is 
[here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643].
 Normally the entire test succeedes in just over 2 minutes.

> DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2
> -
>
> Key: CASSANDRA-13037
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13037
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x
>
>
> DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2:
> http://cassci.datastax.com/job/cassandra-2.2_testall/589/testReport/junit/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/
> http://cassci.datastax.com/job/cassandra-2.1_testall/399/testReport/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/
> {code}
> Error Message
> Timeout occurred. Please note the time in the report does not reflect the 
> time until the timeout.
> Stacktrace
> junit.framework.AssertionFailedError: Timeout occurred. Please note the time 
> in the report does not reflect the time until the timeout.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13037) DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753581#comment-15753581
 ] 

Stefania commented on CASSANDRA-13037:
--

Thanks for checking. I agree that the threads are not deadlocked, my language 
was imprecise. My best guess is that the non-periodic-tasks thread has not been 
signaled, or it has failed to get the correct segment offset before entering 
awaitUninterruptibly(). I'm multiplexing again 
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex/58/], this time 
with a longer timeout (9.5 minutes rather than 5) and with some error log 
messages that are printed out if it takes longer than 1 minute to receive a 
signal in awaitDiskSync(), code is 
[here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643].
 Normally the entire test succeedes in just over 2 minutes.

> DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2
> -
>
> Key: CASSANDRA-13037
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13037
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x
>
>
> DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2:
> http://cassci.datastax.com/job/cassandra-2.2_testall/589/testReport/junit/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/
> http://cassci.datastax.com/job/cassandra-2.1_testall/399/testReport/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/
> {code}
> Error Message
> Timeout occurred. Please note the time in the report does not reflect the 
> time until the timeout.
> Stacktrace
> junit.framework.AssertionFailedError: Timeout occurred. Please note the time 
> in the report does not reflect the time until the timeout.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13049) Too many open files during bootstrapping

[Simon Zhou (JIRA)]

Simon Zhou created CASSANDRA-13049:
--

 Summary: Too many open files during bootstrapping
 Key: CASSANDRA-13049
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13049
 Project: Cassandra
  Issue Type: Improvement
Reporter: Simon Zhou
Assignee: Simon Zhou


We just upgraded from 2.2.5 to 3.0.10 and got issue during bootstrapping. So 
likely this is something made worse along with improving IO performance in 
Cassandra 3.

On our side, the issue is that we have lots of small sstables and thus when 
bootstrapping a new node, it receives lots of files during streaming and 
Cassandra keeps all of them open for an unpredictable amount of time. 
Eventually we hit "Too many open files" error and around that time, I can see 
~1M open files through lsof and almost all of them are *-Data.db and 
*-Index.db. Definitely we should use a better compaction strategy to reduce the 
number of sstables but I see a few possible improvements in Cassandra:

1. We use memory map when reading data from sstables. Every time we create a 
new memory map, there is one more file descriptor open. Memory map improves IO 
performance when dealing with large files, do we want to set a file size 
threshold when doing this?

2. Whenever we finished receiving a file from peer, we create a 
SSTableReader/BigTableReader, which includes opening the data file and index 
file, and keep them open until some time later (unpredictable). See 
StreamReceiveTask#L110, BigTableWriter#openFinal and 
SSTableReader#InstanceTidier. Is it better to lazily open the data/index files 
or close them more often to reclaim the file descriptors?

I searched all known issue in JIRA and looks like this is a new issue in 
Cassandra 3. cc [~Stefania] for comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators

[Ben Bromhead (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Bromhead reassigned CASSANDRA-13048:


Assignee: Ben Bromhead

> Support SASL mechanism negotiation in existing Authenticators
> -
>
> Key: CASSANDRA-13048
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13048
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ben Bromhead
>Assignee: Ben Bromhead
>Priority: Minor
>  Labels: client-auth, client-impacting
>
> [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds 
> support for SASL mechanism negotitation to the native protocol. Existing 
> Authenticators should follow the SASL negotiation mechanism used in 
> [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] .
> It may make sense to make the SASL negotiation mechanism extensible so other 
> Authenticators can use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-12704) snapshot build never be able to publish to mvn artifactory

[Jay Zhuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang reassigned CASSANDRA-12704:
--

Assignee: Jay Zhuang

> snapshot build never be able to publish to mvn artifactory
> --
>
> Key: CASSANDRA-12704
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12704
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.7
>
> Attachments: 12704-trunk.txt
>
>
> {code}
> $ ant publish
> {code}
> works fine when property "release" is set, which publishes the binaries to 
> release Artifactory.
> But for daily snapshot build, if "release" is set, it won't be snapshot build:
> https://github.com/apache/cassandra/blob/cassandra-3.7/build.xml#L74
> if "release" is not set, it doesn't publish to snapshot Artifactory:
> https://github.com/apache/cassandra/blob/cassandra-3.7/build.xml#L1888
> I would suggest just removing the "if check" for target "publish".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10145) Change protocol to allow sending key space independent of query string

[Sandeep Tamhankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752652#comment-15752652
 ] 

Sandeep Tamhankar commented on CASSANDRA-10145:
---

Just confirmed that my branch (PYTHON-684) has been merged to master. We're 
ready to merge that down to cassandra-test and move things forward. I believe 
accepting this patch and merging the Python driver master to cassandra-test 
branch need to happen simultaneously. If the Python driver branch is merged 
down first, it will speak the "new" v5 against existing C* and dtests will fail.

[~thobbs] I think you have the power in both repos, so I defer to you.

> Change protocol to allow sending key space independent of query string
> --
>
> Key: CASSANDRA-10145
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10145
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Vishy Kasar
>Assignee: Sandeep Tamhankar
> Fix For: 3.x
>
> Attachments: 10145-trunk.txt
>
>
> Currently keyspace is either embedded in the query string or set through "use 
> keyspace" on a connection by client driver. 
> There are practical use cases where client user has query and keyspace 
> independently. In order for that scenario to work, they will have to create 
> one client session per keyspace or have to resort to some string replace 
> hackery.
> It will be nice if protocol allowed sending keyspace separately from the 
> query. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11596) Add native transport port to system.peers

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752495#comment-15752495
 ] 

Ariel Weisberg commented on CASSANDRA-11596:


I am doing this as part of CASSANDRA-7544 along with [~rkuris]

> Add native transport port to system.peers
> -
>
> Key: CASSANDRA-11596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11596
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nico Haller
>Assignee: Ariel Weisberg
>Priority: Minor
>  Labels: lhf
>
> Is there any reason why the native transport port is not being stored in 
> system.peers along the rpc broadcast address and transmitted to the connected 
> drivers?
> I would love to have that feature, that would allow me to "hide" my cluster 
> behind a reverse NAT or LB and only consume one external IP address and 
> forward packets based on the port the client is connecting to.
> I guess it makes sense to provide the complete socket information instead of 
> just the address and using a default port setting on the client to complete 
> the connection information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11596) Add native transport port to system.peers

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg reassigned CASSANDRA-11596:
--

Assignee: Ariel Weisberg

> Add native transport port to system.peers
> -
>
> Key: CASSANDRA-11596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11596
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nico Haller
>Assignee: Ariel Weisberg
>Priority: Minor
>  Labels: lhf
>
> Is there any reason why the native transport port is not being stored in 
> system.peers along the rpc broadcast address and transmitted to the connected 
> drivers?
> I would love to have that feature, that would allow me to "hide" my cluster 
> behind a reverse NAT or LB and only consume one external IP address and 
> forward packets based on the port the client is connecting to.
> I guess it makes sense to provide the complete socket information instead of 
> just the address and using a default port setting on the client to complete 
> the connection information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators

[Ben Bromhead (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Bromhead updated CASSANDRA-13048:
-
Issue Type: Sub-task  (was: Task)
Parent: CASSANDRA-9362

> Support SASL mechanism negotiation in existing Authenticators
> -
>
> Key: CASSANDRA-13048
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13048
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ben Bromhead
>Priority: Minor
>  Labels: client-auth, client-impacting
>
> [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds 
> support for SASL mechanism negotitation to the native protocol. Existing 
> Authenticators should follow the SASL negotiation mechanism used in 
> [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] .
> It may make sense to make the SASL negotiation mechanism extensible so other 
> Authenticators can use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators

[Ben Bromhead (JIRA)]

Ben Bromhead created CASSANDRA-13048:


 Summary: Support SASL mechanism negotiation in existing 
Authenticators
 Key: CASSANDRA-13048
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13048
 Project: Cassandra
  Issue Type: Task
Reporter: Ben Bromhead
Priority: Minor


[CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds 
support for SASL mechanism negotitation to the native protocol. Existing 
Authenticators should follow the SASL negotiation mechanism used in 
[CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] .

It may make sense to make the SASL negotiation mechanism extensible so other 
Authenticators can use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12186) anticompaction log message doesn't include the parent repair session id

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-12186:

   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.10
   3.0.11
   Status: Resolved  (was: Ready to Commit)

> anticompaction log message doesn't include the parent repair session id
> ---
>
> Key: CASSANDRA-12186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12186
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Wei Deng
>Assignee: Tommy Stendahl
>Priority: Minor
>  Labels: lhf
> Fix For: 3.0.11, 3.10
>
> Attachments: 12186.txt
>
>
> It appears that even though incremental repair is now enabled by default post 
> C*-3.0 (which means at the end of each repair session, there is an 
> anti-compaction step that needs to be executed), we don't include the parent 
> repair session UUID in the log message of the anti-compaction log entries. 
> This makes observing all activities related to an incremental repair session 
> to be more difficult. See the following:
> {noformat}
> DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956  
> RepairMessageVerbHandler.java:149 - Got anticompaction request 
> AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2}
>  org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4
> <...>
> 
> <...>
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,512  
> CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest 
> on 
> 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')]
>  sstables
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,513  
> CompactionManager.java:540 - SSTable 
> BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')
>  fully contained in range (-9223372036854775808,-9223372036854775808], 
> mutating repairedAt instead of anticompacting
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,570  
> CompactionManager.java:578 - Completed anticompaction successfully
> {noformat}
> The initial submission of the anti-compaction task to the CompactionManager 
> still has reference to the parent repair session UUID, but subsequent 
> anti-compaction log entries are missing this parent repair session UUID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12186) anticompaction log message doesn't include the parent repair session id

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752335#comment-15752335
 ] 

Paulo Motta commented on CASSANDRA-12186:
-

Committed as 2256778726319fb76b6d85c4a47a957116c78147 on 3.0 and merged up. 
Thanks!

> anticompaction log message doesn't include the parent repair session id
> ---
>
> Key: CASSANDRA-12186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12186
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Wei Deng
>Assignee: Tommy Stendahl
>Priority: Minor
>  Labels: lhf
> Fix For: 3.0.11, 3.10
>
> Attachments: 12186.txt
>
>
> It appears that even though incremental repair is now enabled by default post 
> C*-3.0 (which means at the end of each repair session, there is an 
> anti-compaction step that needs to be executed), we don't include the parent 
> repair session UUID in the log message of the anti-compaction log entries. 
> This makes observing all activities related to an incremental repair session 
> to be more difficult. See the following:
> {noformat}
> DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956  
> RepairMessageVerbHandler.java:149 - Got anticompaction request 
> AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2}
>  org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4
> <...>
> 
> <...>
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,512  
> CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest 
> on 
> 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')]
>  sstables
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,513  
> CompactionManager.java:540 - SSTable 
> BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')
>  fully contained in range (-9223372036854775808,-9223372036854775808], 
> mutating repairedAt instead of anticompacting
> INFO  [CompactionExecutor:5] 2016-07-13 02:07:47,570  
> CompactionManager.java:578 - Completed anticompaction successfully
> {noformat}
> The initial submission of the anti-compaction task to the CompactionManager 
> still has reference to the parent repair session UUID, but subsequent 
> anti-compaction log entries are missing this parent repair session UUID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

[paulo]

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68b27b8c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68b27b8c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68b27b8c

Branch: refs/heads/trunk
Commit: 68b27b8c4090f6d04dd0b2f24e54f09e99168cf4
Parents: 7d266b9 fcc19a1
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:55 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:55 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/CHANGES.txt
--
diff --cc CHANGES.txt
index fd888bc1,6cc177c..4506324
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -133,8 -123,9 +133,9 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
+  * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
   * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)
   * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
   * Mark MVs as built after successful bootstrap (CASSANDRA-12984)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/test/unit/org/apache/cassandra/db/compaction/AntiCompactionTest.java
--

[2/4] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d

Branch: refs/heads/trunk
Commit: 9924756da9c248c2365448998e63a36708188260
Parents: 489be96 2256778
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:12 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:12 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt
--
diff --cc CHANGES.txt
index fa0c94a,a40dabd..562b63d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without secondary 

[1/4] cassandra git commit: Add parent repair session id to anticompaction log message

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 7d266b9e7 -> 68b27b8c4


Add parent repair session id to anticompaction log message

Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787

Branch: refs/heads/trunk
Commit: 2256778726319fb76b6d85c4a47a957116c78147
Parents: 48abc03
Author: tommy stendahl 
Authored: Tue Nov 15 14:52:57 2016 +0100
Committer: Paulo Motta 
Committed: Thu Dec 15 17:38:20 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 63e095d..a40dabd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add parent repair session id to anticompaction log message (CASSANDRA-12186)
  * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a77cefb..28140e0 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -456,7 +456,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 public ListenableFuture submitAntiCompaction(final ColumnFamilyStore 
cfs,
   final Collection 
ranges,
   final Refs sstables,
-  final long repairedAt)
+  final long repairedAt,
+  final UUID parentRepairSession)
 {
 Runnable runnable = new WrappedRunnable() {
 @Override
@@ -475,7 +476,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 sstables.release(compactedSSTables);
 modifier = cfs.getTracker().tryModify(sstables, 
OperationType.ANTICOMPACTION);
 }
-performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt);
+performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt, parentRepairSession);
 }
 };
 
@@ -500,6 +501,7 @@ public class CompactionManager implements 
CompactionManagerMBean
  * @param cfs
  * @param ranges Ranges that the repair was carried out on
  * @param validatedForRepair SSTables containing the repaired ranges. 
Should be referenced before passing them.
+ * @param parentRepairSession parent repair session ID
  * @throws InterruptedException
  * @throws IOException
  */
@@ -507,10 +509,11 @@ public class CompactionManager implements 
CompactionManagerMBean
   Collection ranges,
   Refs validatedForRepair,
   LifecycleTransaction txn,
-  long repairedAt) throws 
InterruptedException, IOException
+  long repairedAt,
+  UUID parentRepairSession) throws 
InterruptedException, IOException
 {
-logger.info("Starting anticompaction for {}.{} on {}/{} sstables", 
cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), 
cfs.getLiveSSTables());
-logger.trace("Starting anticompaction for ranges {}", ranges);
+logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} 
sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), 
validatedForRepair.size(), cfs.getLiveSSTables());
+logger.trace("[repair #{}] Starting anticompaction for ranges {}", 
parentRepairSession, ranges);
 Set sstables = new 

[3/4] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

[paulo]

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fcc19a1f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fcc19a1f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fcc19a1f

Branch: refs/heads/trunk
Commit: fcc19a1fa022dec02b8fe0c94e10973202c7bd50
Parents: b1e23ab 9924756
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:37 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:37 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

[paulo]

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fcc19a1f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fcc19a1f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fcc19a1f

Branch: refs/heads/cassandra-3.X
Commit: fcc19a1fa022dec02b8fe0c94e10973202c7bd50
Parents: b1e23ab 9924756
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:37 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:37 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

[1/3] cassandra git commit: Add parent repair session id to anticompaction log message

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.X b1e23ab54 -> fcc19a1fa


Add parent repair session id to anticompaction log message

Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787

Branch: refs/heads/cassandra-3.X
Commit: 2256778726319fb76b6d85c4a47a957116c78147
Parents: 48abc03
Author: tommy stendahl 
Authored: Tue Nov 15 14:52:57 2016 +0100
Committer: Paulo Motta 
Committed: Thu Dec 15 17:38:20 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 63e095d..a40dabd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add parent repair session id to anticompaction log message (CASSANDRA-12186)
  * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a77cefb..28140e0 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -456,7 +456,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 public ListenableFuture submitAntiCompaction(final ColumnFamilyStore 
cfs,
   final Collection 
ranges,
   final Refs sstables,
-  final long repairedAt)
+  final long repairedAt,
+  final UUID parentRepairSession)
 {
 Runnable runnable = new WrappedRunnable() {
 @Override
@@ -475,7 +476,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 sstables.release(compactedSSTables);
 modifier = cfs.getTracker().tryModify(sstables, 
OperationType.ANTICOMPACTION);
 }
-performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt);
+performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt, parentRepairSession);
 }
 };
 
@@ -500,6 +501,7 @@ public class CompactionManager implements 
CompactionManagerMBean
  * @param cfs
  * @param ranges Ranges that the repair was carried out on
  * @param validatedForRepair SSTables containing the repaired ranges. 
Should be referenced before passing them.
+ * @param parentRepairSession parent repair session ID
  * @throws InterruptedException
  * @throws IOException
  */
@@ -507,10 +509,11 @@ public class CompactionManager implements 
CompactionManagerMBean
   Collection ranges,
   Refs validatedForRepair,
   LifecycleTransaction txn,
-  long repairedAt) throws 
InterruptedException, IOException
+  long repairedAt,
+  UUID parentRepairSession) throws 
InterruptedException, IOException
 {
-logger.info("Starting anticompaction for {}.{} on {}/{} sstables", 
cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), 
cfs.getLiveSSTables());
-logger.trace("Starting anticompaction for ranges {}", ranges);
+logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} 
sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), 
validatedForRepair.size(), cfs.getLiveSSTables());
+logger.trace("[repair #{}] Starting anticompaction for ranges {}", 
parentRepairSession, ranges);
 

[2/3] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d

Branch: refs/heads/cassandra-3.X
Commit: 9924756da9c248c2365448998e63a36708188260
Parents: 489be96 2256778
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:12 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:12 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt
--
diff --cc CHANGES.txt
index fa0c94a,a40dabd..562b63d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without 

[1/2] cassandra git commit: Add parent repair session id to anticompaction log message

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 489be961c -> 9924756da


Add parent repair session id to anticompaction log message

Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787

Branch: refs/heads/cassandra-3.11
Commit: 2256778726319fb76b6d85c4a47a957116c78147
Parents: 48abc03
Author: tommy stendahl 
Authored: Tue Nov 15 14:52:57 2016 +0100
Committer: Paulo Motta 
Committed: Thu Dec 15 17:38:20 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 63e095d..a40dabd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add parent repair session id to anticompaction log message (CASSANDRA-12186)
  * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a77cefb..28140e0 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -456,7 +456,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 public ListenableFuture submitAntiCompaction(final ColumnFamilyStore 
cfs,
   final Collection 
ranges,
   final Refs sstables,
-  final long repairedAt)
+  final long repairedAt,
+  final UUID parentRepairSession)
 {
 Runnable runnable = new WrappedRunnable() {
 @Override
@@ -475,7 +476,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 sstables.release(compactedSSTables);
 modifier = cfs.getTracker().tryModify(sstables, 
OperationType.ANTICOMPACTION);
 }
-performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt);
+performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt, parentRepairSession);
 }
 };
 
@@ -500,6 +501,7 @@ public class CompactionManager implements 
CompactionManagerMBean
  * @param cfs
  * @param ranges Ranges that the repair was carried out on
  * @param validatedForRepair SSTables containing the repaired ranges. 
Should be referenced before passing them.
+ * @param parentRepairSession parent repair session ID
  * @throws InterruptedException
  * @throws IOException
  */
@@ -507,10 +509,11 @@ public class CompactionManager implements 
CompactionManagerMBean
   Collection ranges,
   Refs validatedForRepair,
   LifecycleTransaction txn,
-  long repairedAt) throws 
InterruptedException, IOException
+  long repairedAt,
+  UUID parentRepairSession) throws 
InterruptedException, IOException
 {
-logger.info("Starting anticompaction for {}.{} on {}/{} sstables", 
cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), 
cfs.getLiveSSTables());
-logger.trace("Starting anticompaction for ranges {}", ranges);
+logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} 
sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), 
validatedForRepair.size(), cfs.getLiveSSTables());
+logger.trace("[repair #{}] Starting anticompaction for ranges {}", 
parentRepairSession, ranges);
 

[2/2] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d

Branch: refs/heads/cassandra-3.11
Commit: 9924756da9c248c2365448998e63a36708188260
Parents: 489be96 2256778
Author: Paulo Motta 
Authored: Thu Dec 15 17:40:12 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 17:40:12 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt
--
diff --cc CHANGES.txt
index fa0c94a,a40dabd..562b63d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without 

cassandra git commit: Add parent repair session id to anticompaction log message

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 48abc0369 -> 225677872


Add parent repair session id to anticompaction log message

Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787

Branch: refs/heads/cassandra-3.0
Commit: 2256778726319fb76b6d85c4a47a957116c78147
Parents: 48abc03
Author: tommy stendahl 
Authored: Tue Nov 15 14:52:57 2016 +0100
Committer: Paulo Motta 
Committed: Thu Dec 15 17:38:20 2016 -0200

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 21 +++-
 .../cassandra/service/ActiveRepairService.java  |  4 ++--
 .../db/compaction/AntiCompactionTest.java   | 17 ++--
 4 files changed, 26 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 63e095d..a40dabd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add parent repair session id to anticompaction log message (CASSANDRA-12186)
  * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a77cefb..28140e0 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -456,7 +456,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 public ListenableFuture submitAntiCompaction(final ColumnFamilyStore 
cfs,
   final Collection 
ranges,
   final Refs sstables,
-  final long repairedAt)
+  final long repairedAt,
+  final UUID parentRepairSession)
 {
 Runnable runnable = new WrappedRunnable() {
 @Override
@@ -475,7 +476,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 sstables.release(compactedSSTables);
 modifier = cfs.getTracker().tryModify(sstables, 
OperationType.ANTICOMPACTION);
 }
-performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt);
+performAnticompaction(cfs, ranges, sstables, modifier, 
repairedAt, parentRepairSession);
 }
 };
 
@@ -500,6 +501,7 @@ public class CompactionManager implements 
CompactionManagerMBean
  * @param cfs
  * @param ranges Ranges that the repair was carried out on
  * @param validatedForRepair SSTables containing the repaired ranges. 
Should be referenced before passing them.
+ * @param parentRepairSession parent repair session ID
  * @throws InterruptedException
  * @throws IOException
  */
@@ -507,10 +509,11 @@ public class CompactionManager implements 
CompactionManagerMBean
   Collection ranges,
   Refs validatedForRepair,
   LifecycleTransaction txn,
-  long repairedAt) throws 
InterruptedException, IOException
+  long repairedAt,
+  UUID parentRepairSession) throws 
InterruptedException, IOException
 {
-logger.info("Starting anticompaction for {}.{} on {}/{} sstables", 
cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), 
cfs.getLiveSSTables());
-logger.trace("Starting anticompaction for ranges {}", ranges);
+logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} 
sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), 
validatedForRepair.size(), cfs.getLiveSSTables());
+logger.trace("[repair #{}] Starting anticompaction for ranges {}", 
parentRepairSession, ranges);
 

[jira] [Updated] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-12905:

   Resolution: Fixed
Fix Version/s: 3.0.11
   Status: Resolved  (was: Patch Available)

bq. If we have a hint file that is being processed and a WTE occurs in the 
middle, will the whole file be retransmitted or can it be resumed at the last 
successful position?

It will resume at the latest successful page, not from the start of the hint 
file.

bq. I guess this is not the case from my personal observations. I had 
situations with > 1GB hint queues per sender node which were not going to 
shrink due to WTEs. It seemed like the same hints have been retransmitted over 
and over again from scratch instead of trying to resume. 

Perhaps the hard-coded hint page size of 512KB is too coarse grained for 
cluster with MVs, what causes too much data to be re-transmitted on retry 
generating even more contention. We should perhaps make this configurable.

bq. What helped in this situation was to pause hint delivery on all nodes but 
1-2. I'm pretty sure the problem was WTEs due to lock contentions. To be 
honest, I did not try lowering hinted_handoff_throttle_in_kb or at least I 
don't remember.

Yeah, lowering {{hinted_handoff_throttle_in_kb}} should help for clusters with 
MVs, and perhaps tunable hint page sizes too. You could maybe experiment with 
different values to see if you reach a more stable state.

I committed the current work to 3.0 and up as 
3faa0d925791be085b92949a0a0ec20f7e6ae368 and 
48abc0369799acca0521150e2c88d4032e01c3b5 so we can unblock 3.10 release. Feel 
free to open follow-up tickets to further improve MV hints and streaming with 
the points we discussed here.

Thanks!

> Retry acquire MV lock on failure instead of throwing WTE on streaming
> -
>
> Key: CASSANDRA-12905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: centos 6.7 x86_64
>Reporter: Nir Zilka
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.11, 3.10
>
>
> Hello,
> I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, 
> private VLAN),
> first it was 2.2.5.1 and repair worked flawlessly,
> second upgrade was to 3.0.9 (with upgradesstables) and also repair worked 
> well,
> then i upgraded 2 weeks ago to 3.9 - and the repair problems started.
> there are several errors types from the system.log (different nodes) :
> - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
> - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation 
> timed out - received only 0 responses
> - Remote peer xxx.xxx.xxx.xxx failed stream session
> - Session completed with the following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> 
> i use 3.9 default configuration with the cluster settings adjustments (3 
> seeds, GossipingPropertyFileSnitch).
> streaming_socket_timeout_in_ms is the default (8640).
> i'm afraid from consistency problems while i'm not performing repair.
> Any ideas?
> Thanks,
> Nir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra-builds git commit: Fix triggerInterval variable scope

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master dce81a6f3 -> 32939b643


Fix triggerInterval variable scope


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/32939b64
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/32939b64
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/32939b64

Branch: refs/heads/master
Commit: 32939b643e22cc609fccae5d8d02f35cb754da02
Parents: dce81a6
Author: Michael Shuler 
Authored: Thu Dec 15 13:19:44 2016 -0600
Committer: Michael Shuler 
Committed: Thu Dec 15 13:19:44 2016 -0600

--
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/32939b64/jenkins-dsl/cassandra_job_dsl_seed.groovy
--
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index be5bb0e..311f5e7 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -226,9 +226,9 @@ cassandraBranches.each {
 
 // Run default dtest daily and variations weekly
 if (targetName == 'dtest') {
-def triggerInterval = '@daily'
+triggerInterval = '@daily'
 } else {
-def triggerInterval = '@weekly'
+triggerInterval = '@weekly'
 }
 
 // Skip dtest-offheap on cassandra-3.0 branch
@@ -242,7 +242,7 @@ cassandraBranches.each {
 node / scm / branches / 'hudson.plugins.git.BranchSpec' / 
name(branchName)
 }
 triggers {
-scm("${triggerInterval}")
+scm(triggerInterval)
 }
 steps {
 shell("./cassandra-builds/build-scripts/cassandra-dtest.sh 
${targetName}")

[4/5] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

[paulo]

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b1e23ab5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b1e23ab5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b1e23ab5

Branch: refs/heads/trunk
Commit: b1e23ab54f659755fdd7c1adf5c14b05088337f6
Parents: e5a77cf 489be96
Author: Paulo Motta 
Authored: Thu Dec 15 16:50:30 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:50:30 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b1e23ab5/CHANGES.txt
--

[2/5] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock

[paulo]

Make hint delivery async so MV hint is deferred on failure to acquire lock

Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036

Branch: refs/heads/trunk
Commit: 48abc0369799acca0521150e2c88d4032e01c3b5
Parents: 3faa0d9
Author: Paulo Motta 
Authored: Thu Dec 15 12:49:38 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:05 2016 -0200

--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++--
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 2 files changed, 30 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java
--
diff --git a/src/java/org/apache/cassandra/hints/Hint.java 
b/src/java/org/apache/cassandra/hints/Hint.java
index cbb5e74..17fbf5d 100644
--- a/src/java/org/apache/cassandra/hints/Hint.java
+++ b/src/java/org/apache/cassandra/hints/Hint.java
@@ -19,8 +19,12 @@ package org.apache.cassandra.hints;
 
 import java.io.IOException;
 import java.util.*;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
+import com.google.common.base.Throwables;
+
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.io.IVersionedSerializer;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -68,7 +72,7 @@ public final class Hint
 return new Hint(mutation, creationTime, mutation.smallestGCGS());
 }
 
-/**
+/*
  * @param mutation the hinted mutation
  * @param creationTime time of this hint's creation (in milliseconds since 
epoch)
  * @param gcgs the smallest gcgs of all tables involved at the time of 
hint creation (in seconds)
@@ -81,19 +85,33 @@ public final class Hint
 /**
  * Applies the contained mutation unless it's expired, filtering out any 
updates for truncated tables
  */
-void apply()
+CompletableFuture applyFuture()
 {
-if (!isLive())
-return;
+if (isLive())
+{
+// filter out partition update for table that have been truncated 
since hint's creation
+Mutation filtered = mutation;
+for (UUID id : mutation.getColumnFamilyIds())
+if (creationTime <= SystemKeyspace.getTruncatedAt(id))
+filtered = filtered.without(id);
 
-// filter out partition update for table that have been truncated 
since hint's creation
-Mutation filtered = mutation;
-for (UUID id : mutation.getColumnFamilyIds())
-if (creationTime <= SystemKeyspace.getTruncatedAt(id))
-filtered = filtered.without(id);
+if (!filtered.isEmpty())
+return filtered.applyFuture();
+}
+
+return CompletableFuture.completedFuture(null);
+}
 
-if (!filtered.isEmpty())
-filtered.apply();
+void apply()
+{
+try
+{
+applyFuture().get();
+}
+catch (Exception e)
+{
+throw Throwables.propagate(e.getCause());
+}
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index d8838a9..abcd1f9 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -88,10 +88,8 @@ public final class HintVerbHandler implements 
IVerbHandler
 else
 {
 // the common path - the node is both the destination and a valid 
replica for the hint.
-hint.apply();
+hint.applyFuture().thenAccept(o -> reply(id, 
message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); 
return null;});
 }
-
-reply(id, message.from);
 }
 
 private static void reply(int id, InetAddress to)

[1/5] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 0f67b540e -> 7d266b9e7


Retry acquire MV lock on failure instead of throwing WTE on streaming

Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92

Branch: refs/heads/trunk
Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368
Parents: 9fc1ffb
Author: brstgt 
Authored: Thu Dec 15 12:42:31 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:00 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 92 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  5 +-
 6 files changed, 63 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e69bf08..63e095d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/Keyspace.java 
b/src/java/org/apache/cassandra/db/Keyspace.java
index ec5102b..3715995 100644
--- a/src/java/org/apache/cassandra/db/Keyspace.java
+++ b/src/java/org/apache/cassandra/db/Keyspace.java
@@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock;
 
 import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
@@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.JVMStabilityInspector;
 import org.apache.cassandra.utils.concurrent.OpOrder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /**
  * It represents a Keyspace.
@@ -62,7 +62,7 @@ public class Keyspace
 
 private static final String TEST_FAIL_WRITES_KS = 
System.getProperty("cassandra.test.fail_writes_ks", "");
 private static final boolean TEST_FAIL_WRITES = 
!TEST_FAIL_WRITES_KS.isEmpty();
-private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", 
"0"), 0);
+private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger("cassandra.test.fail_mv_locks_count", 0);
 
 public final KeyspaceMetrics metric;
 
@@ -379,42 +379,40 @@ public class Keyspace
 }
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog)
-{
-return apply(mutation, writeCommitLog, true, false, null);
-}
-
-/**
- * Should be used if caller is blocking and runs in mutation stage.
- * Otherwise there is a race condition where ALL mutation workers are 
beeing blocked ending
- * in a complete deadlock of the mutation stage. See CASSANDRA-12689.
- *
- * @param mutation
- * @param writeCommitLog
- * @return
- */
-public CompletableFuture applyNotDeferrable(Mutation mutation, boolean 
writeCommitLog)
+public CompletableFuture applyFuture(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
 {
-return apply(mutation, writeCommitLog, true, false, false, null);
+return apply(mutation, writeCommitLog, updateIndexes, true, true, 
null);
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
+public void apply(Mutation mutation, boolean writeCommitLog, boolean 
updateIndexes)
 {
-return apply(mutation, writeCommitLog, updateIndexes, false, null);
+apply(mutation, writeCommitLog, updateIndexes, true);
 }
 
-public CompletableFuture applyFromCommitLog(Mutation mutation)
+public void apply(final Mutation mutation,
+  final boolean writeCommitLog)

[5/5] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

[paulo]

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d266b9e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d266b9e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d266b9e

Branch: refs/heads/trunk
Commit: 7d266b9e79e0738df25e4f86b5bdccb030a02d88
Parents: 0f67b54 b1e23ab
Author: Paulo Motta 
Authored: Thu Dec 15 16:51:12 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:51:12 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d266b9e/CHANGES.txt
--
diff --cc CHANGES.txt
index e0c5637,5f7cf93..fd888bc1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -133,8 -123,9 +133,9 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
+  * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)
   * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
   * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
   * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d266b9e/src/java/org/apache/cassandra/db/Mutation.java
--

[3/5] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961

Branch: refs/heads/trunk
Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54
Parents: 73547a3 48abc03
Author: Paulo Motta 
Authored: Thu Dec 15 16:46:32 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:49:10 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt
--
diff --cc CHANGES.txt
index 3db0179,63e095d..fa0c94a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 

[4/4] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

[paulo]

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b1e23ab5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b1e23ab5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b1e23ab5

Branch: refs/heads/cassandra-3.X
Commit: b1e23ab54f659755fdd7c1adf5c14b05088337f6
Parents: e5a77cf 489be96
Author: Paulo Motta 
Authored: Thu Dec 15 16:50:30 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:50:30 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b1e23ab5/CHANGES.txt
--

[1/4] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.X e5a77cfea -> b1e23ab54


Retry acquire MV lock on failure instead of throwing WTE on streaming

Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92

Branch: refs/heads/cassandra-3.X
Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368
Parents: 9fc1ffb
Author: brstgt 
Authored: Thu Dec 15 12:42:31 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:00 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 92 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  5 +-
 6 files changed, 63 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e69bf08..63e095d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/Keyspace.java 
b/src/java/org/apache/cassandra/db/Keyspace.java
index ec5102b..3715995 100644
--- a/src/java/org/apache/cassandra/db/Keyspace.java
+++ b/src/java/org/apache/cassandra/db/Keyspace.java
@@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock;
 
 import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
@@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.JVMStabilityInspector;
 import org.apache.cassandra.utils.concurrent.OpOrder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /**
  * It represents a Keyspace.
@@ -62,7 +62,7 @@ public class Keyspace
 
 private static final String TEST_FAIL_WRITES_KS = 
System.getProperty("cassandra.test.fail_writes_ks", "");
 private static final boolean TEST_FAIL_WRITES = 
!TEST_FAIL_WRITES_KS.isEmpty();
-private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", 
"0"), 0);
+private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger("cassandra.test.fail_mv_locks_count", 0);
 
 public final KeyspaceMetrics metric;
 
@@ -379,42 +379,40 @@ public class Keyspace
 }
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog)
-{
-return apply(mutation, writeCommitLog, true, false, null);
-}
-
-/**
- * Should be used if caller is blocking and runs in mutation stage.
- * Otherwise there is a race condition where ALL mutation workers are 
beeing blocked ending
- * in a complete deadlock of the mutation stage. See CASSANDRA-12689.
- *
- * @param mutation
- * @param writeCommitLog
- * @return
- */
-public CompletableFuture applyNotDeferrable(Mutation mutation, boolean 
writeCommitLog)
+public CompletableFuture applyFuture(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
 {
-return apply(mutation, writeCommitLog, true, false, false, null);
+return apply(mutation, writeCommitLog, updateIndexes, true, true, 
null);
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
+public void apply(Mutation mutation, boolean writeCommitLog, boolean 
updateIndexes)
 {
-return apply(mutation, writeCommitLog, updateIndexes, false, null);
+apply(mutation, writeCommitLog, updateIndexes, true);
 }
 
-public CompletableFuture applyFromCommitLog(Mutation mutation)
+public void apply(final Mutation mutation,
+  final boolean 

[3/4] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961

Branch: refs/heads/cassandra-3.X
Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54
Parents: 73547a3 48abc03
Author: Paulo Motta 
Authored: Thu Dec 15 16:46:32 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:49:10 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt
--
diff --cc CHANGES.txt
index 3db0179,63e095d..fa0c94a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the 

[2/4] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock

[paulo]

Make hint delivery async so MV hint is deferred on failure to acquire lock

Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036

Branch: refs/heads/cassandra-3.X
Commit: 48abc0369799acca0521150e2c88d4032e01c3b5
Parents: 3faa0d9
Author: Paulo Motta 
Authored: Thu Dec 15 12:49:38 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:05 2016 -0200

--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++--
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 2 files changed, 30 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java
--
diff --git a/src/java/org/apache/cassandra/hints/Hint.java 
b/src/java/org/apache/cassandra/hints/Hint.java
index cbb5e74..17fbf5d 100644
--- a/src/java/org/apache/cassandra/hints/Hint.java
+++ b/src/java/org/apache/cassandra/hints/Hint.java
@@ -19,8 +19,12 @@ package org.apache.cassandra.hints;
 
 import java.io.IOException;
 import java.util.*;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
+import com.google.common.base.Throwables;
+
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.io.IVersionedSerializer;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -68,7 +72,7 @@ public final class Hint
 return new Hint(mutation, creationTime, mutation.smallestGCGS());
 }
 
-/**
+/*
  * @param mutation the hinted mutation
  * @param creationTime time of this hint's creation (in milliseconds since 
epoch)
  * @param gcgs the smallest gcgs of all tables involved at the time of 
hint creation (in seconds)
@@ -81,19 +85,33 @@ public final class Hint
 /**
  * Applies the contained mutation unless it's expired, filtering out any 
updates for truncated tables
  */
-void apply()
+CompletableFuture applyFuture()
 {
-if (!isLive())
-return;
+if (isLive())
+{
+// filter out partition update for table that have been truncated 
since hint's creation
+Mutation filtered = mutation;
+for (UUID id : mutation.getColumnFamilyIds())
+if (creationTime <= SystemKeyspace.getTruncatedAt(id))
+filtered = filtered.without(id);
 
-// filter out partition update for table that have been truncated 
since hint's creation
-Mutation filtered = mutation;
-for (UUID id : mutation.getColumnFamilyIds())
-if (creationTime <= SystemKeyspace.getTruncatedAt(id))
-filtered = filtered.without(id);
+if (!filtered.isEmpty())
+return filtered.applyFuture();
+}
+
+return CompletableFuture.completedFuture(null);
+}
 
-if (!filtered.isEmpty())
-filtered.apply();
+void apply()
+{
+try
+{
+applyFuture().get();
+}
+catch (Exception e)
+{
+throw Throwables.propagate(e.getCause());
+}
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index d8838a9..abcd1f9 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -88,10 +88,8 @@ public final class HintVerbHandler implements 
IVerbHandler
 else
 {
 // the common path - the node is both the destination and a valid 
replica for the hint.
-hint.apply();
+hint.applyFuture().thenAccept(o -> reply(id, 
message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); 
return null;});
 }
-
-reply(id, message.from);
 }
 
 private static void reply(int id, InetAddress to)

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[paulo]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961

Branch: refs/heads/cassandra-3.11
Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54
Parents: 73547a3 48abc03
Author: Paulo Motta 
Authored: Thu Dec 15 16:46:32 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:49:10 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 89 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++---
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  6 +-
 8 files changed, 90 insertions(+), 86 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt
--
diff --cc CHANGES.txt
index 3db0179,63e095d..fa0c94a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the 

[2/3] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock

[paulo]

Make hint delivery async so MV hint is deferred on failure to acquire lock

Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036

Branch: refs/heads/cassandra-3.11
Commit: 48abc0369799acca0521150e2c88d4032e01c3b5
Parents: 3faa0d9
Author: Paulo Motta 
Authored: Thu Dec 15 12:49:38 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:05 2016 -0200

--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++--
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 2 files changed, 30 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java
--
diff --git a/src/java/org/apache/cassandra/hints/Hint.java 
b/src/java/org/apache/cassandra/hints/Hint.java
index cbb5e74..17fbf5d 100644
--- a/src/java/org/apache/cassandra/hints/Hint.java
+++ b/src/java/org/apache/cassandra/hints/Hint.java
@@ -19,8 +19,12 @@ package org.apache.cassandra.hints;
 
 import java.io.IOException;
 import java.util.*;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
+import com.google.common.base.Throwables;
+
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.io.IVersionedSerializer;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -68,7 +72,7 @@ public final class Hint
 return new Hint(mutation, creationTime, mutation.smallestGCGS());
 }
 
-/**
+/*
  * @param mutation the hinted mutation
  * @param creationTime time of this hint's creation (in milliseconds since 
epoch)
  * @param gcgs the smallest gcgs of all tables involved at the time of 
hint creation (in seconds)
@@ -81,19 +85,33 @@ public final class Hint
 /**
  * Applies the contained mutation unless it's expired, filtering out any 
updates for truncated tables
  */
-void apply()
+CompletableFuture applyFuture()
 {
-if (!isLive())
-return;
+if (isLive())
+{
+// filter out partition update for table that have been truncated 
since hint's creation
+Mutation filtered = mutation;
+for (UUID id : mutation.getColumnFamilyIds())
+if (creationTime <= SystemKeyspace.getTruncatedAt(id))
+filtered = filtered.without(id);
 
-// filter out partition update for table that have been truncated 
since hint's creation
-Mutation filtered = mutation;
-for (UUID id : mutation.getColumnFamilyIds())
-if (creationTime <= SystemKeyspace.getTruncatedAt(id))
-filtered = filtered.without(id);
+if (!filtered.isEmpty())
+return filtered.applyFuture();
+}
+
+return CompletableFuture.completedFuture(null);
+}
 
-if (!filtered.isEmpty())
-filtered.apply();
+void apply()
+{
+try
+{
+applyFuture().get();
+}
+catch (Exception e)
+{
+throw Throwables.propagate(e.getCause());
+}
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index d8838a9..abcd1f9 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -88,10 +88,8 @@ public final class HintVerbHandler implements 
IVerbHandler
 else
 {
 // the common path - the node is both the destination and a valid 
replica for the hint.
-hint.apply();
+hint.applyFuture().thenAccept(o -> reply(id, 
message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); 
return null;});
 }
-
-reply(id, message.from);
 }
 
 private static void reply(int id, InetAddress to)

[1/3] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 73547a342 -> 489be961c


Retry acquire MV lock on failure instead of throwing WTE on streaming

Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92

Branch: refs/heads/cassandra-3.11
Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368
Parents: 9fc1ffb
Author: brstgt 
Authored: Thu Dec 15 12:42:31 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:00 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 92 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  5 +-
 6 files changed, 63 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e69bf08..63e095d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/Keyspace.java 
b/src/java/org/apache/cassandra/db/Keyspace.java
index ec5102b..3715995 100644
--- a/src/java/org/apache/cassandra/db/Keyspace.java
+++ b/src/java/org/apache/cassandra/db/Keyspace.java
@@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock;
 
 import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
@@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.JVMStabilityInspector;
 import org.apache.cassandra.utils.concurrent.OpOrder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /**
  * It represents a Keyspace.
@@ -62,7 +62,7 @@ public class Keyspace
 
 private static final String TEST_FAIL_WRITES_KS = 
System.getProperty("cassandra.test.fail_writes_ks", "");
 private static final boolean TEST_FAIL_WRITES = 
!TEST_FAIL_WRITES_KS.isEmpty();
-private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", 
"0"), 0);
+private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger("cassandra.test.fail_mv_locks_count", 0);
 
 public final KeyspaceMetrics metric;
 
@@ -379,42 +379,40 @@ public class Keyspace
 }
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog)
-{
-return apply(mutation, writeCommitLog, true, false, null);
-}
-
-/**
- * Should be used if caller is blocking and runs in mutation stage.
- * Otherwise there is a race condition where ALL mutation workers are 
beeing blocked ending
- * in a complete deadlock of the mutation stage. See CASSANDRA-12689.
- *
- * @param mutation
- * @param writeCommitLog
- * @return
- */
-public CompletableFuture applyNotDeferrable(Mutation mutation, boolean 
writeCommitLog)
+public CompletableFuture applyFuture(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
 {
-return apply(mutation, writeCommitLog, true, false, false, null);
+return apply(mutation, writeCommitLog, updateIndexes, true, true, 
null);
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
+public void apply(Mutation mutation, boolean writeCommitLog, boolean 
updateIndexes)
 {
-return apply(mutation, writeCommitLog, updateIndexes, false, null);
+apply(mutation, writeCommitLog, updateIndexes, true);
 }
 
-public CompletableFuture applyFromCommitLog(Mutation mutation)
+public void apply(final Mutation mutation,
+  final boolean 

[2/2] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock

[paulo]

Make hint delivery async so MV hint is deferred on failure to acquire lock

Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036

Branch: refs/heads/cassandra-3.0
Commit: 48abc0369799acca0521150e2c88d4032e01c3b5
Parents: 3faa0d9
Author: Paulo Motta 
Authored: Thu Dec 15 12:49:38 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:05 2016 -0200

--
 src/java/org/apache/cassandra/hints/Hint.java   | 40 ++--
 .../apache/cassandra/hints/HintVerbHandler.java |  4 +-
 2 files changed, 30 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java
--
diff --git a/src/java/org/apache/cassandra/hints/Hint.java 
b/src/java/org/apache/cassandra/hints/Hint.java
index cbb5e74..17fbf5d 100644
--- a/src/java/org/apache/cassandra/hints/Hint.java
+++ b/src/java/org/apache/cassandra/hints/Hint.java
@@ -19,8 +19,12 @@ package org.apache.cassandra.hints;
 
 import java.io.IOException;
 import java.util.*;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
+import com.google.common.base.Throwables;
+
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.io.IVersionedSerializer;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -68,7 +72,7 @@ public final class Hint
 return new Hint(mutation, creationTime, mutation.smallestGCGS());
 }
 
-/**
+/*
  * @param mutation the hinted mutation
  * @param creationTime time of this hint's creation (in milliseconds since 
epoch)
  * @param gcgs the smallest gcgs of all tables involved at the time of 
hint creation (in seconds)
@@ -81,19 +85,33 @@ public final class Hint
 /**
  * Applies the contained mutation unless it's expired, filtering out any 
updates for truncated tables
  */
-void apply()
+CompletableFuture applyFuture()
 {
-if (!isLive())
-return;
+if (isLive())
+{
+// filter out partition update for table that have been truncated 
since hint's creation
+Mutation filtered = mutation;
+for (UUID id : mutation.getColumnFamilyIds())
+if (creationTime <= SystemKeyspace.getTruncatedAt(id))
+filtered = filtered.without(id);
 
-// filter out partition update for table that have been truncated 
since hint's creation
-Mutation filtered = mutation;
-for (UUID id : mutation.getColumnFamilyIds())
-if (creationTime <= SystemKeyspace.getTruncatedAt(id))
-filtered = filtered.without(id);
+if (!filtered.isEmpty())
+return filtered.applyFuture();
+}
+
+return CompletableFuture.completedFuture(null);
+}
 
-if (!filtered.isEmpty())
-filtered.apply();
+void apply()
+{
+try
+{
+applyFuture().get();
+}
+catch (Exception e)
+{
+throw Throwables.propagate(e.getCause());
+}
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index d8838a9..abcd1f9 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -88,10 +88,8 @@ public final class HintVerbHandler implements 
IVerbHandler
 else
 {
 // the common path - the node is both the destination and a valid 
replica for the hint.
-hint.apply();
+hint.applyFuture().thenAccept(o -> reply(id, 
message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); 
return null;});
 }
-
-reply(id, message.from);
 }
 
 private static void reply(int id, InetAddress to)

[1/2] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 9fc1ffb63 -> 48abc0369


Retry acquire MV lock on failure instead of throwing WTE on streaming

Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92

Branch: refs/heads/cassandra-3.0
Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368
Parents: 9fc1ffb
Author: brstgt 
Authored: Thu Dec 15 12:42:31 2016 -0200
Committer: Paulo Motta 
Committed: Thu Dec 15 16:46:00 2016 -0200

--
 CHANGES.txt |  1 +
 src/java/org/apache/cassandra/db/Keyspace.java  | 92 +++-
 src/java/org/apache/cassandra/db/Mutation.java  | 17 ++--
 .../db/commitlog/CommitLogReplayer.java | 10 +--
 .../cassandra/service/paxos/PaxosState.java |  9 +-
 .../cassandra/streaming/StreamReceiveTask.java  |  5 +-
 6 files changed, 63 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e69bf08..63e095d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Improve contention handling on failure to acquire MV lock for streaming and 
hints (CASSANDRA-12905)
  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/Keyspace.java 
b/src/java/org/apache/cassandra/db/Keyspace.java
index ec5102b..3715995 100644
--- a/src/java/org/apache/cassandra/db/Keyspace.java
+++ b/src/java/org/apache/cassandra/db/Keyspace.java
@@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock;
 
 import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
@@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.JVMStabilityInspector;
 import org.apache.cassandra.utils.concurrent.OpOrder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /**
  * It represents a Keyspace.
@@ -62,7 +62,7 @@ public class Keyspace
 
 private static final String TEST_FAIL_WRITES_KS = 
System.getProperty("cassandra.test.fail_writes_ks", "");
 private static final boolean TEST_FAIL_WRITES = 
!TEST_FAIL_WRITES_KS.isEmpty();
-private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", 
"0"), 0);
+private static int TEST_FAIL_MV_LOCKS_COUNT = 
Integer.getInteger("cassandra.test.fail_mv_locks_count", 0);
 
 public final KeyspaceMetrics metric;
 
@@ -379,42 +379,40 @@ public class Keyspace
 }
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog)
-{
-return apply(mutation, writeCommitLog, true, false, null);
-}
-
-/**
- * Should be used if caller is blocking and runs in mutation stage.
- * Otherwise there is a race condition where ALL mutation workers are 
beeing blocked ending
- * in a complete deadlock of the mutation stage. See CASSANDRA-12689.
- *
- * @param mutation
- * @param writeCommitLog
- * @return
- */
-public CompletableFuture applyNotDeferrable(Mutation mutation, boolean 
writeCommitLog)
+public CompletableFuture applyFuture(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
 {
-return apply(mutation, writeCommitLog, true, false, false, null);
+return apply(mutation, writeCommitLog, updateIndexes, true, true, 
null);
 }
 
-public CompletableFuture apply(Mutation mutation, boolean 
writeCommitLog, boolean updateIndexes)
+public void apply(Mutation mutation, boolean writeCommitLog, boolean 
updateIndexes)
 {
-return apply(mutation, writeCommitLog, updateIndexes, false, null);
+apply(mutation, writeCommitLog, updateIndexes, true);
 }
 
-public CompletableFuture applyFromCommitLog(Mutation mutation)
+public void apply(final Mutation mutation,
+  final boolean 

[jira] [Updated] (CASSANDRA-12510) Disallow decommission when number of replicas will drop below configured RF

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-12510:
-
Attachment: 12510-3.x-v2.patch

Attached an updated version of the patch.

Switched it back to cloneAfterAllLeft as I think I understand this a bit better 
now and there is no reason for settled.
Also switched the {{isMember}} check back to the current tokenMetadata rather 
than the clone, this was just my misunderstanding of how things work before.
And made the changes regarding the LEAVING state. I ended up including the 
whole for loop in the check (pending range check included) as I'm pretty sure 
it makes sense that a node can't have pending ranges while it's in the leaving 
state.

[~pauloricardomg] I re-ran most of the failing tests and they seem OK now, not 
sure if I potentially broke other things though if you want to have a look :)

> Disallow decommission when number of replicas will drop below configured RF
> ---
>
> Key: CASSANDRA-12510
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12510
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
> Environment: C* version 3.3
>Reporter: Atin Sood
>Assignee: Kurt Greaves
>Priority: Minor
>  Labels: lhf
> Attachments: 12510-3.x-v2.patch, 12510-3.x.patch
>
>
> Steps to replicate :
> - Create a 3 node cluster in DC1 and create a keyspace test_keyspace with 
> table test_table with replication strategy NetworkTopologyStrategy , DC1=3 . 
> Populate some data into this table.
> - Add 5 more nodes to this cluster, but in DC2. Also do not alter the 
> keyspace to add the new DC2 to replication (this is intentional and the 
> reason why the bug shows up). So the desc keyspace should still list 
> NetworkTopologyStrategy with DC1=3 as RF
> - As expected, this will now be a 8 node cluster with 3 nodes in DC1 and 5 in 
> DC2
> - Now start decommissioning the nodes in DC1. Note that the decommission runs 
> fine on all the 3 nodes, but since the new nodes are in DC2 and the RF for 
> keyspace is restricted to DC1, the new 5 nodes won't get any data.
> - You will now end with the 5 node cluster which has no data from the 
> decommissioned 3 nodes and hence ending up in data loss
> I do understand that this problem could have been avoided if we perform an 
> alter stmt and add DC2 replication before adding the 5 nodes. But the fact 
> that decommission ran fine on the 3 nodes on DC1 without complaining that 
> there were no nodes to stream its data seems a little discomforting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra-builds git commit: Pass DSL a varable string for interval

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master 2a86477d5 -> dce81a6f3


Pass DSL a varable string for interval

ERROR: (cassandra_job_dsl_seed.groovy, line 245) No such property: 
triggerInterval for class: javaposse.jobdsl.dsl.helpers.triggers.TriggerContext


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/dce81a6f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/dce81a6f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/dce81a6f

Branch: refs/heads/master
Commit: dce81a6f380559b13f3df48f350db6c82b3cfa11
Parents: 2a86477
Author: Michael Shuler 
Authored: Thu Dec 15 13:01:15 2016 -0600
Committer: Michael Shuler 
Committed: Thu Dec 15 13:01:15 2016 -0600

--
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/dce81a6f/jenkins-dsl/cassandra_job_dsl_seed.groovy
--
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 7bb30a4..be5bb0e 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -242,7 +242,7 @@ cassandraBranches.each {
 node / scm / branches / 'hudson.plugins.git.BranchSpec' / 
name(branchName)
 }
 triggers {
-scm(triggerInterval)
+scm("${triggerInterval}")
 }
 steps {
 shell("./cassandra-builds/build-scripts/cassandra-dtest.sh 
${targetName}")

cassandra-builds git commit: Run default dtest daily and variations weekly

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master bb0105561 -> 2a86477d5


Run default dtest daily and variations weekly


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/2a86477d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/2a86477d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/2a86477d

Branch: refs/heads/master
Commit: 2a86477d5f7f73e6e9cd17468ea10679f078d81a
Parents: bb01055
Author: Michael Shuler 
Authored: Thu Dec 15 12:46:33 2016 -0600
Committer: Michael Shuler 
Committed: Thu Dec 15 12:46:33 2016 -0600

--
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/2a86477d/jenkins-dsl/cassandra_job_dsl_seed.groovy
--
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 02bf485..7bb30a4 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -150,9 +150,6 @@ job('Cassandra-template-dtest') {
 }
 }
 }
-triggers {
-scm('@daily')
-}
 steps {
 buildDescription('', buildDescStr)
 shell("git clean -xdff ; git clone ${buildsRepo} ; git clone 
${dtestRepo}")
@@ -227,6 +224,13 @@ cassandraBranches.each {
 dtestTargets.each {
 def targetName = it
 
+// Run default dtest daily and variations weekly
+if (targetName == 'dtest') {
+def triggerInterval = '@daily'
+} else {
+def triggerInterval = '@weekly'
+}
+
 // Skip dtest-offheap on cassandra-3.0 branch
 if ((targetName == 'dtest-offheap') && (branchName == 
'cassandra-3.0')) {
 println("Skipping ${targetName} on branch ${branchName}")
@@ -237,6 +241,9 @@ cassandraBranches.each {
 configure { node ->
 node / scm / branches / 'hudson.plugins.git.BranchSpec' / 
name(branchName)
 }
+triggers {
+scm(triggerInterval)
+}
 steps {
 shell("./cassandra-builds/build-scripts/cassandra-dtest.sh 
${targetName}")
 }

[jira] [Commented] (CASSANDRA-12947) Repair not replicating data

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752033#comment-15752033
 ] 

Jeremiah Jordan commented on CASSANDRA-12947:
-

If you had to run scrub to fix some tables, did you run a repair -full after 
doing that?  Scrub is going to throw away data, that has possibly already been 
repaired, so you need to do a repair -full to get it back.

> Repair not replicating data
> ---
>
> Key: CASSANDRA-12947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12947
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.7
> Single DC
> 7 Nodes
> RF 3
> NetworkTopologyStrategy
> OS: Ubuntu
>Reporter: Malte Pickhan
> Fix For: 3.x
>
>
> We experienced strange behaviour of our C* cluster last week.
> In the logs we've seen multiple requests in the logs where we the values read 
> from Cassandra actually have been 'null'. 
> When running the query on a node, we sometimes received the result and 
> sometimes not.
> When setting the CL to LOCAL_QUORUM everything was fine. So far nothing 
> unusual, probably the dataset wasn't replicated to one of the nodes.
> When turning on the tracing and running the query there was following 
> intersting line:
> {quote}
> Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 |   
> $PUBLIC_IP |126 | 127.0.0.1
>  Digest mismatch: org.apache.cassandra.service.DigestMismatchException: 
> Mismatch for key DecoratedKey(-5887526567589486157, 
> 3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs 
> d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
> {quote}
> This is probably related to CASSANDRA-12090?
> The interesting part is, after that we've run a 'nodetool repair -pr', after 
> that the behaviour was still the same and the data randomly not available, 
> depending on which node was hit.
> Only after running a 'nodetool repair -pr -full' the issue was gone.
> Did we miss something here? The point that's bothering me is that the dataset 
> was not replicated.
> Worth to note is probably that some weeks ago we've hit the bug 
> CASSANDRA-12694 and fixed it by scrubbing some tables.
> Any hints/help are appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-12829:
---
   Resolution: Fixed
Fix Version/s: 3.10
   3.0.11
   Status: Resolved  (was: Patch Available)

> DELETE query with an empty IN clause can delete more than expected
> --
>
> Key: CASSANDRA-12829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12829
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded 
> from the website
>Reporter: Jason T. Bradshaw
>Assignee: Alex Petrov
> Fix For: 3.0.11, 3.10
>
>
> When deleting from a table with a certain structure and using an *in* clause 
> with an empty list, the *in* clause with an empty list can be ignored, 
> resulting in deleting more than is expected.
> *Setup:*
> {code}
> cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id));
> cqlsh> insert into test (a, b, id) values ('a', 'b', 
> ----);
> cqlsh> insert into test (a, b, id) values ('b', 'c', 
> ----);
> cqlsh> insert into test (a, b, id) values ('a', 'c', 
> ----);
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Expected:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Actual:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  b | c | ----
> (1 rows)
> {code}
> Instead of deleting nothing, as the final empty *in* clause would imply, it 
> instead deletes everything that matches the first two clauses, acting as if 
> the following query had been issued instead:
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c');
> {code}
> This seems to be related to the presence of a tuple clustering key, as I 
> could not reproduce it without one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751913#comment-15751913
 ] 

Benjamin Lerer commented on CASSANDRA-12829:


Thanks for the patch.

Committed into 3.0 at 9fc1ffb631a5a44bc11b55061288758e39a9d20a and merged into, 
3.11, 3.X and trunk

> DELETE query with an empty IN clause can delete more than expected
> --
>
> Key: CASSANDRA-12829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12829
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded 
> from the website
>Reporter: Jason T. Bradshaw
>Assignee: Alex Petrov
>
> When deleting from a table with a certain structure and using an *in* clause 
> with an empty list, the *in* clause with an empty list can be ignored, 
> resulting in deleting more than is expected.
> *Setup:*
> {code}
> cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id));
> cqlsh> insert into test (a, b, id) values ('a', 'b', 
> ----);
> cqlsh> insert into test (a, b, id) values ('b', 'c', 
> ----);
> cqlsh> insert into test (a, b, id) values ('a', 'c', 
> ----);
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Expected:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Actual:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  b | c | ----
> (1 rows)
> {code}
> Instead of deleting nothing, as the final empty *in* clause would imply, it 
> instead deletes everything that matches the first two clauses, acting as if 
> the following query had been issued instead:
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c');
> {code}
> This seems to be related to the presence of a tuple clustering key, as I 
> could not reproduce it without one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11

[blerer]

Merge branch cassandra-3.0 into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34

Branch: refs/heads/trunk
Commit: 73547a3421cc92e07d0373846fc1bff796570275
Parents: 249e4b1 9fc1ffb
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:04:34 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:07:59 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt
--
diff --cc CHANGES.txt
index f95dd81,e69bf08..3db0179
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies 

[4/4] cassandra git commit: Merge branch cassandra-3.X into trunk

[blerer]

Merge branch cassandra-3.X into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0f67b540
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0f67b540
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0f67b540

Branch: refs/heads/trunk
Commit: 0f67b540e8e2c022348e8c163879d433e8d18fc3
Parents: b88ddb8 e5a77cf
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:14:44 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:14:56 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/CHANGES.txt
--
diff --cc CHANGES.txt
index 9e7dd80,d32d91c..e0c5637
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -133,8 -123,9 +133,9 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
+  * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
   * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
   * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
   * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
--

[1/4] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk b88ddb8ab -> 0f67b540e


Fix DELETE and UPDATE queries with empty IN restrictions

patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6

Branch: refs/heads/trunk
Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a
Parents: 0fe82be
Author: Alex Petrov 
Authored: Thu Dec 15 17:56:41 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:01:32 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 
 .../cql3/statements/BatchStatement.java | 11 ++--
 .../cql3/statements/ModificationStatement.java  | 15 +++--
 .../org/apache/cassandra/cql3/CQLTester.java|  7 ++
 .../cql3/validation/operations/BatchTest.java   | 40 
 .../cql3/validation/operations/DeleteTest.java  | 63 --
 .../cql3/validation/operations/InsertTest.java  |  6 --
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 69 +++-
 10 files changed, 202 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 145afb9..e69bf08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
  * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 647d22f..542dec9 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -430,6 +430,17 @@ public final class StatementRestrictions
 }
 
 /**
+ * Checks if restrictions on the clustering key have IN restrictions.
+ *
+ * @return true if the restrictions on the clustering key 
have IN restrictions,
+ * false otherwise.
+ */
+public boolean clusteringKeyRestrictionsHasIN()
+{
+return clusteringColumnsRestrictions.isIN();
+}
+
+/**
  * Processes the clustering column restrictions.
  *
  * @param hasQueriableIndex true if some of the queried data 
are indexed, false otherwise

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index f0aa835..76a6460 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement
 QueryOptions statementOptions = options.forStatement(i);
 long timestamp = attrs.getTimestamp(now, statementOptions);
 List pks = 
statement.buildPartitionKeyNames(statementOptions);
-if (pks.size() > 1)
+if (statement.getRestrictions().keyIsInRelation())
 throw new IllegalArgumentException("Batch with conditions 
cannot span multiple partitions (you cannot use IN on the partition key)");
 if (key == null)
 {
@@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement
 throw new InvalidRequestException("Batch with conditions 
cannot span multiple partitions");
 }
 
-SortedSet clusterings = 
statement.createClustering(statementOptions);
+
checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(),
+   "IN on the clustering key columns is not supported with 
conditional %s",
+   statement.type.isUpdate()? "updates" : "deletions");
 
-

[3/4] cassandra git commit: Merge branch cassandra-3.11 into cassandra-3.X

[blerer]

Merge branch cassandra-3.11 into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e5a77cfe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e5a77cfe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e5a77cfe

Branch: refs/heads/trunk
Commit: e5a77cfeaa2be7f6c68965603e7a87034ff7b576
Parents: de84f26 73547a3
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:09:52 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:10:01 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--

[3/3] cassandra git commit: Merge branch cassandra-3.11 into cassandra-3.X

[blerer]

Merge branch cassandra-3.11 into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e5a77cfe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e5a77cfe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e5a77cfe

Branch: refs/heads/cassandra-3.X
Commit: e5a77cfeaa2be7f6c68965603e7a87034ff7b576
Parents: de84f26 73547a3
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:09:52 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:10:01 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--

[1/3] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.X de84f264f -> e5a77cfea


Fix DELETE and UPDATE queries with empty IN restrictions

patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6

Branch: refs/heads/cassandra-3.X
Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a
Parents: 0fe82be
Author: Alex Petrov 
Authored: Thu Dec 15 17:56:41 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:01:32 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 
 .../cql3/statements/BatchStatement.java | 11 ++--
 .../cql3/statements/ModificationStatement.java  | 15 +++--
 .../org/apache/cassandra/cql3/CQLTester.java|  7 ++
 .../cql3/validation/operations/BatchTest.java   | 40 
 .../cql3/validation/operations/DeleteTest.java  | 63 --
 .../cql3/validation/operations/InsertTest.java  |  6 --
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 69 +++-
 10 files changed, 202 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 145afb9..e69bf08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
  * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 647d22f..542dec9 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -430,6 +430,17 @@ public final class StatementRestrictions
 }
 
 /**
+ * Checks if restrictions on the clustering key have IN restrictions.
+ *
+ * @return true if the restrictions on the clustering key 
have IN restrictions,
+ * false otherwise.
+ */
+public boolean clusteringKeyRestrictionsHasIN()
+{
+return clusteringColumnsRestrictions.isIN();
+}
+
+/**
  * Processes the clustering column restrictions.
  *
  * @param hasQueriableIndex true if some of the queried data 
are indexed, false otherwise

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index f0aa835..76a6460 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement
 QueryOptions statementOptions = options.forStatement(i);
 long timestamp = attrs.getTimestamp(now, statementOptions);
 List pks = 
statement.buildPartitionKeyNames(statementOptions);
-if (pks.size() > 1)
+if (statement.getRestrictions().keyIsInRelation())
 throw new IllegalArgumentException("Batch with conditions 
cannot span multiple partitions (you cannot use IN on the partition key)");
 if (key == null)
 {
@@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement
 throw new InvalidRequestException("Batch with conditions 
cannot span multiple partitions");
 }
 
-SortedSet clusterings = 
statement.createClustering(statementOptions);
+
checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(),
+   "IN on the clustering key columns is not supported with 
conditional %s",
+   statement.type.isUpdate()? "updates" : "deletions");
 
-

[2/3] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11

[blerer]

Merge branch cassandra-3.0 into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34

Branch: refs/heads/cassandra-3.X
Commit: 73547a3421cc92e07d0373846fc1bff796570275
Parents: 249e4b1 9fc1ffb
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:04:34 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:07:59 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt
--
diff --cc CHANGES.txt
index f95dd81,e69bf08..3db0179
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure 

[1/2] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 249e4b195 -> 73547a342


Fix DELETE and UPDATE queries with empty IN restrictions

patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6

Branch: refs/heads/cassandra-3.11
Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a
Parents: 0fe82be
Author: Alex Petrov 
Authored: Thu Dec 15 17:56:41 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:01:32 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 
 .../cql3/statements/BatchStatement.java | 11 ++--
 .../cql3/statements/ModificationStatement.java  | 15 +++--
 .../org/apache/cassandra/cql3/CQLTester.java|  7 ++
 .../cql3/validation/operations/BatchTest.java   | 40 
 .../cql3/validation/operations/DeleteTest.java  | 63 --
 .../cql3/validation/operations/InsertTest.java  |  6 --
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 69 +++-
 10 files changed, 202 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 145afb9..e69bf08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
  * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 647d22f..542dec9 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -430,6 +430,17 @@ public final class StatementRestrictions
 }
 
 /**
+ * Checks if restrictions on the clustering key have IN restrictions.
+ *
+ * @return true if the restrictions on the clustering key 
have IN restrictions,
+ * false otherwise.
+ */
+public boolean clusteringKeyRestrictionsHasIN()
+{
+return clusteringColumnsRestrictions.isIN();
+}
+
+/**
  * Processes the clustering column restrictions.
  *
  * @param hasQueriableIndex true if some of the queried data 
are indexed, false otherwise

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index f0aa835..76a6460 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement
 QueryOptions statementOptions = options.forStatement(i);
 long timestamp = attrs.getTimestamp(now, statementOptions);
 List pks = 
statement.buildPartitionKeyNames(statementOptions);
-if (pks.size() > 1)
+if (statement.getRestrictions().keyIsInRelation())
 throw new IllegalArgumentException("Batch with conditions 
cannot span multiple partitions (you cannot use IN on the partition key)");
 if (key == null)
 {
@@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement
 throw new InvalidRequestException("Batch with conditions 
cannot span multiple partitions");
 }
 
-SortedSet clusterings = 
statement.createClustering(statementOptions);
+
checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(),
+   "IN on the clustering key columns is not supported with 
conditional %s",
+   statement.type.isUpdate()? "updates" : "deletions");
 
-

[2/2] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11

[blerer]

Merge branch cassandra-3.0 into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34

Branch: refs/heads/cassandra-3.11
Commit: 73547a3421cc92e07d0373846fc1bff796570275
Parents: 249e4b1 9fc1ffb
Author: Benjamin Lerer 
Authored: Thu Dec 15 18:04:34 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:07:59 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 +++
 .../cql3/statements/BatchStatement.java | 11 ++-
 .../cql3/statements/ModificationStatement.java  | 10 ++-
 .../cql3/validation/operations/BatchTest.java   | 40 +++
 .../cql3/validation/operations/DeleteTest.java  | 57 
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 70 
 8 files changed, 194 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt
--
diff --cc CHANGES.txt
index f95dd81,e69bf08..3db0179
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,114 -1,5 +1,115 @@@
 -3.0.11
 +3.10
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure 

cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 0fe82be83 -> 9fc1ffb63


Fix DELETE and UPDATE queries with empty IN restrictions

patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6

Branch: refs/heads/cassandra-3.0
Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a
Parents: 0fe82be
Author: Alex Petrov 
Authored: Thu Dec 15 17:56:41 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Dec 15 18:01:32 2016 +0100

--
 CHANGES.txt |  1 +
 .../restrictions/StatementRestrictions.java | 11 
 .../cql3/statements/BatchStatement.java | 11 ++--
 .../cql3/statements/ModificationStatement.java  | 15 +++--
 .../org/apache/cassandra/cql3/CQLTester.java|  7 ++
 .../cql3/validation/operations/BatchTest.java   | 40 
 .../cql3/validation/operations/DeleteTest.java  | 63 --
 .../cql3/validation/operations/InsertTest.java  |  6 --
 .../operations/InsertUpdateIfConditionTest.java |  6 ++
 .../cql3/validation/operations/UpdateTest.java  | 69 +++-
 10 files changed, 202 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 145afb9..e69bf08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
  * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
  * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
  * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 647d22f..542dec9 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -430,6 +430,17 @@ public final class StatementRestrictions
 }
 
 /**
+ * Checks if restrictions on the clustering key have IN restrictions.
+ *
+ * @return true if the restrictions on the clustering key 
have IN restrictions,
+ * false otherwise.
+ */
+public boolean clusteringKeyRestrictionsHasIN()
+{
+return clusteringColumnsRestrictions.isIN();
+}
+
+/**
  * Processes the clustering column restrictions.
  *
  * @param hasQueriableIndex true if some of the queried data 
are indexed, false otherwise

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index f0aa835..76a6460 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement
 QueryOptions statementOptions = options.forStatement(i);
 long timestamp = attrs.getTimestamp(now, statementOptions);
 List pks = 
statement.buildPartitionKeyNames(statementOptions);
-if (pks.size() > 1)
+if (statement.getRestrictions().keyIsInRelation())
 throw new IllegalArgumentException("Batch with conditions 
cannot span multiple partitions (you cannot use IN on the partition key)");
 if (key == null)
 {
@@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement
 throw new InvalidRequestException("Batch with conditions 
cannot span multiple partitions");
 }
 
-SortedSet clusterings = 
statement.createClustering(statementOptions);
+
checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(),
+   "IN on the clustering key columns is not supported with 
conditional %s",
+   statement.type.isUpdate()? "updates" : "deletions");
 
-

[jira] [Commented] (CASSANDRA-5051) Allow automatic cleanup after gc_grace

[Jon Haddad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751853#comment-15751853
 ] 

Jon Haddad commented on CASSANDRA-5051:
---

Maybe time for this in 4.0?

> Allow automatic cleanup after gc_grace
> --
>
> Key: CASSANDRA-5051
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5051
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Brandon Williams
>Assignee: Vijay
>  Labels: vnodes
> Fix For: 3.x
>
> Attachments: 0001-5051-v4.patch, 0001-5051-v6.patch, 
> 0001-5051-with-test-fixes.patch, 0001-CASSANDRA-5051.patch, 
> 0002-5051-remove-upgradesstable-v4.patch, 
> 0002-5051-remove-upgradesstable.patch, 0004-5051-additional-test-v4.patch, 
> 5051-v2.txt
>
>
> When using vnodes, after adding a new node you have to run cleanup on all the 
> machines, because you don't know which are affected and chances are it was 
> most if not all of them.  As an alternative to this intensive process, we 
> could allow cleanup during compaction if the data is older than gc_grace (or 
> perhaps some other time period since people tend to use gc_grace hacks to get 
> rid of tombstones.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751847#comment-15751847
 ] 

Benjamin Lerer commented on CASSANDRA-12829:


+1

> DELETE query with an empty IN clause can delete more than expected
> --
>
> Key: CASSANDRA-12829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12829
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded 
> from the website
>Reporter: Jason T. Bradshaw
>Assignee: Alex Petrov
>
> When deleting from a table with a certain structure and using an *in* clause 
> with an empty list, the *in* clause with an empty list can be ignored, 
> resulting in deleting more than is expected.
> *Setup:*
> {code}
> cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id));
> cqlsh> insert into test (a, b, id) values ('a', 'b', 
> ----);
> cqlsh> insert into test (a, b, id) values ('b', 'c', 
> ----);
> cqlsh> insert into test (a, b, id) values ('a', 'c', 
> ----);
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Expected:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  a | c | ----
>  b | c | ----
>  a | b | ----
> (3 rows)
> {code}
> *Actual:*
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in ();
> cqlsh> select * from test;
>  a | b | id
> ---+---+--
>  b | c | ----
> (1 rows)
> {code}
> Instead of deleting nothing, as the final empty *in* clause would imply, it 
> instead deletes everything that matches the first two clauses, acting as if 
> the following query had been issued instead:
> {code}
> cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c');
> {code}
> This seems to be related to the presence of a tuple clustering key, as I 
> could not reproduce it without one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12859) Column-level permissions

[Boris Melamed (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751746#comment-15751746
 ] 

Boris Melamed commented on CASSANDRA-12859:
---

Thank you Stefan.
Since that public commit in my fork, I've written more edge case dtests.
Some of them discovered a bug which I've fixed (when accessing columns, some of 
which are granted on different roles of the user). To be published on my fork.
I'm currently adding more tests around MVs.

Is there a spec for what's the expected behavior with permissions and MVs?
There are not many dtests for those, and from reading the code, it seems quite 
different from RDBMS view permissions.
For example, in C*:
- No permissions are required on an MV for SELECT. Only its base table must be 
granted.
- For modifying a base table, the user must have permissions on all its MVs. 
This is starkly different from RDBMS, IMHO.
I can try and proceed likewise for columns, but would like to make sure this is 
expected.

> Column-level permissions
> 
>
> Key: CASSANDRA-12859
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12859
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core, CQL
>Reporter: Boris Melamed
>  Labels: doc-impacting
> Attachments: Cassandra Proposal - Column-level permissions v2.docx, 
> Cassandra Proposal - Column-level permissions.docx
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> h4. Here is a draft of: 
> Cassandra Proposal - Column-level permissions.docx (attached)
> h4. Quoting the 'Overview' section:
> The purpose of this proposal is to add column-level (field-level) permissions 
> to Cassandra. It is my intent to soon start implementing this feature in a 
> fork, and to submit a pull request once it’s ready.
> h4. Motivation
> Cassandra already supports permissions on keyspace and table (column family) 
> level. Sources:
> * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra
> * https://cassandra.apache.org/doc/latest/cql/security.html#data-control
> At IBM, we have use cases in the area of big data analytics where 
> column-level access permissions are also a requirement. All industry RDBMS 
> products are supporting this level of permission control, and regulators are 
> expecting it from all data-based systems.
> h4. Main day-one requirements
> # Extend CQL (Cassandra Query Language) to be able to optionally specify a 
> list of individual columns, in the {{GRANT}} statement. The relevant 
> permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and 
> {{SELECT}}.
> # Persist the optional information in the appropriate system table 
> ‘system_auth.role_permissions’.
> # Enforce the column access restrictions during execution. Details:
> #* Should fit with the existing permission propagation down a role chain.
> #* Proposed message format when a user’s roles give access to the queried 
> table but not to all of the selected, inserted, or updated columns:
>   "User %s has no %s permission on column %s of table %s"
> #* Error will report only the first checked column. 
> Nice to have: list all inaccessible columns.
> #* Error code is the same as for table access denial: 2100.
> h4. Additional day-one requirements
> # Reflect the column-level permissions in statements of type 
> {{LIST ALL PERMISSIONS OF someuser;}}
> # When columns are dropped or renamed, trigger purging or adapting of their 
> permissions
> # Performance should not degrade in any significant way.
> # Backwards compatibility
> #* Permission enforcement for DBs created before the upgrade should continue 
> to work with the same behavior after upgrading to a version that allows 
> column-level permissions.
> #* Previous CQL syntax will remain valid, and have the same effect as before.
> h4. Documentation
> * 
> https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission
> * Feedback request: any others?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13047) Point cqlsh help to the new doc

[Sylvain Lebresne (JIRA)]

Sylvain Lebresne created CASSANDRA-13047:


 Summary: Point cqlsh help to the new doc
 Key: CASSANDRA-13047
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13047
 Project: Cassandra
  Issue Type: Bug
Reporter: Sylvain Lebresne
 Fix For: 4.0


Cqlsh still points to the "old" textile CQL doc, but that's not really maintain 
anymore now that we have the new doc (which include everything the old doc had 
and more). We should update cqlsh to point to the new doc so we can remove the 
old one completely.

Any takers?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Daniel Miranda (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749713#comment-15749713
 ] 

Daniel Miranda edited comment on CASSANDRA-11678 at 12/15/16 3:44 PM:
--

I am observing this issue in 3.9.0 without any external crashes or reboots 
being involved. I'm running a 3-node test cluster in AWS.

I have KairosDB [1] writing synthetic data from a stress program into the 
cluster. After a few minutes of stress one or more nodes crash with corrupted 
hint files. After the initial crash, I wiped one of the nodes clean and 
rebootstrapped, and in the next attempt the other two nodes crashed.

I have hints_compression set to LZ4, and I can send a copy of the logs and 
corrupted hint files if they are of interest.

[1]: http://kairosdb.github.io


was (Author: danielkza):
I am observing this issue in 3.9.0 without any external crashes or reboots 
being involved. I'm running a 3-node test cluster in AWS.

I have KairosDB [1] writing synthetic data from a stress program into the 
cluster. Affter a few minutes of stress one or more nodes crash with corrupted 
hint files. After the initial crash, I wiped one of the nodes clean and 
rebootstrapped, and in the next attempt the other two nodes crashed.

I have hints_compression set to LZ4, and I can send a copy of the logs and 
corrupted hint files if they are of interest.

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception in V 3.5.0 *
> {code}
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> 

[jira] [Commented] (CASSANDRA-12859) Column-level permissions

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751678#comment-15751678
 ] 

Sylvain Lebresne commented on CASSANDRA-12859:
--

bq.  There's {{doc/cql3/CQL.textile}} ({{ant generate-cql-html}})

Don't bother, this is basically deprecated and should really have gone away 
already.

bq. and {{doc/source/cql/security.rst}} that could use some hints how this works

{{ant gen-doc}}

> Column-level permissions
> 
>
> Key: CASSANDRA-12859
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12859
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core, CQL
>Reporter: Boris Melamed
>  Labels: doc-impacting
> Attachments: Cassandra Proposal - Column-level permissions v2.docx, 
> Cassandra Proposal - Column-level permissions.docx
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> h4. Here is a draft of: 
> Cassandra Proposal - Column-level permissions.docx (attached)
> h4. Quoting the 'Overview' section:
> The purpose of this proposal is to add column-level (field-level) permissions 
> to Cassandra. It is my intent to soon start implementing this feature in a 
> fork, and to submit a pull request once it’s ready.
> h4. Motivation
> Cassandra already supports permissions on keyspace and table (column family) 
> level. Sources:
> * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra
> * https://cassandra.apache.org/doc/latest/cql/security.html#data-control
> At IBM, we have use cases in the area of big data analytics where 
> column-level access permissions are also a requirement. All industry RDBMS 
> products are supporting this level of permission control, and regulators are 
> expecting it from all data-based systems.
> h4. Main day-one requirements
> # Extend CQL (Cassandra Query Language) to be able to optionally specify a 
> list of individual columns, in the {{GRANT}} statement. The relevant 
> permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and 
> {{SELECT}}.
> # Persist the optional information in the appropriate system table 
> ‘system_auth.role_permissions’.
> # Enforce the column access restrictions during execution. Details:
> #* Should fit with the existing permission propagation down a role chain.
> #* Proposed message format when a user’s roles give access to the queried 
> table but not to all of the selected, inserted, or updated columns:
>   "User %s has no %s permission on column %s of table %s"
> #* Error will report only the first checked column. 
> Nice to have: list all inaccessible columns.
> #* Error code is the same as for table access denial: 2100.
> h4. Additional day-one requirements
> # Reflect the column-level permissions in statements of type 
> {{LIST ALL PERMISSIONS OF someuser;}}
> # When columns are dropped or renamed, trigger purging or adapting of their 
> permissions
> # Performance should not degrade in any significant way.
> # Backwards compatibility
> #* Permission enforcement for DBs created before the upgrade should continue 
> to work with the same behavior after upgrading to a version that allows 
> column-level permissions.
> #* Previous CQL syntax will remain valid, and have the same effect as before.
> h4. Documentation
> * 
> https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission
> * Feedback request: any others?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12859) Column-level permissions

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751659#comment-15751659
 ] 

Stefan Podkowinski commented on CASSANDRA-12859:


This looks really promising. Just played around with your branch and it worked 
as expected for me.

Can you also update the documentation for syntax changes and examples? There's 
{{doc/cql3/CQL.textile}} ({{ant generate-cql-html}}) and 
{{doc/source/cql/security.rst}} that could use some hints how this works. 


> Column-level permissions
> 
>
> Key: CASSANDRA-12859
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12859
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core, CQL
>Reporter: Boris Melamed
>  Labels: doc-impacting
> Attachments: Cassandra Proposal - Column-level permissions v2.docx, 
> Cassandra Proposal - Column-level permissions.docx
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> h4. Here is a draft of: 
> Cassandra Proposal - Column-level permissions.docx (attached)
> h4. Quoting the 'Overview' section:
> The purpose of this proposal is to add column-level (field-level) permissions 
> to Cassandra. It is my intent to soon start implementing this feature in a 
> fork, and to submit a pull request once it’s ready.
> h4. Motivation
> Cassandra already supports permissions on keyspace and table (column family) 
> level. Sources:
> * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra
> * https://cassandra.apache.org/doc/latest/cql/security.html#data-control
> At IBM, we have use cases in the area of big data analytics where 
> column-level access permissions are also a requirement. All industry RDBMS 
> products are supporting this level of permission control, and regulators are 
> expecting it from all data-based systems.
> h4. Main day-one requirements
> # Extend CQL (Cassandra Query Language) to be able to optionally specify a 
> list of individual columns, in the {{GRANT}} statement. The relevant 
> permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and 
> {{SELECT}}.
> # Persist the optional information in the appropriate system table 
> ‘system_auth.role_permissions’.
> # Enforce the column access restrictions during execution. Details:
> #* Should fit with the existing permission propagation down a role chain.
> #* Proposed message format when a user’s roles give access to the queried 
> table but not to all of the selected, inserted, or updated columns:
>   "User %s has no %s permission on column %s of table %s"
> #* Error will report only the first checked column. 
> Nice to have: list all inaccessible columns.
> #* Error code is the same as for table access denial: 2100.
> h4. Additional day-one requirements
> # Reflect the column-level permissions in statements of type 
> {{LIST ALL PERMISSIONS OF someuser;}}
> # When columns are dropped or renamed, trigger purging or adapting of their 
> permissions
> # Performance should not degrade in any significant way.
> # Backwards compatibility
> #* Permission enforcement for DBs created before the upgrade should continue 
> to work with the same behavior after upgrading to a version that allows 
> column-level permissions.
> #* Previous CQL syntax will remain valid, and have the same effect as before.
> h4. Documentation
> * 
> https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission
> * Feedback request: any others?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11995) Commitlog replaced with all NULs

[John Marsten (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644485#comment-15644485
 ] 

John Marsten edited comment on CASSANDRA-11995 at 12/15/16 3:31 PM:


We observed this phenomenon running Apache Cassandra 3.7 on Windows Server 2012 
VM. The precise history and state of the VM in question was unclear, but 
apparently the administrators took a live volume snapshot, and after reverting 
to the snapshot, Cassandra would no longer start due to the error above. In 
this case, the commit log file was 48MB of NUL bytes. The Cassandra system log 
showed an ungraceful termination and a handful of warnings related to slow 
commit log syncs, but no errors prior to the CommitLogReplayException on 
subsequent startups.

Cassandra was configured for batch mode commit log syncing with a window size 
of 50ms. It's not clear whether the system was writing to Cassandra at the time 
of the snapshot.


was (Author: jmarsten):
We observed this phenomenon running Apache Cassandra 3.8 on Windows Server 2012 
VM. The precise history and state of the VM in question was unclear, but 
apparently the administrators took a live snapshot, and after reverting to the 
snapshot, Cassandra would no longer start due to the error above. In this case, 
the commit log file was 48MB of NUL bytes. The Cassandra system log showed an 
ungraceful termination and a handful of warnings related to slow commit log 
syncs, but no errors prior to the CommitLogReplayException on subsequent 
startups.

Cassandra was configured for batch mode commit log syncing with a window size 
of 50ms. It's not clear whether the system was writing to Cassandra at the time 
of the snapshot.

> Commitlog replaced with all NULs
> 
>
> Key: CASSANDRA-11995
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11995
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Windows 10 Enterprise 1511
> DataStax Cassandra Community Server 2.2.3
>Reporter: James Howe
>
> I noticed this morning that Cassandra was failing to start, after being shut 
> down on Friday.
> {code}
> ERROR 09:13:37 Exiting due to error while processing commit log during 
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Could not read commit log descriptor in file C:\Program Files\DataStax 
> Community\data\commitlog\CommitLog-5-1465571056722.log
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) 
> [apache-cassandra-2.2.3.jar:2.2.3]
> {code}
> Checking the referenced file reveals it comprises 33,554,432 (32 * 1024 * 
> 1024) NUL bytes.
> No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and 
> appear exactly as normal.
> Is installed as a service via DataStax's distribution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13046) Missing PID file directory when installing from RPM

[Marcel Dopita (JIRA)]

Marcel Dopita created CASSANDRA-13046:
-

 Summary: Missing PID file directory when installing from RPM
 Key: CASSANDRA-13046
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13046
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
 Environment: CentOS 7.3
Reporter: Marcel Dopita
Priority: Minor


As already reported in CASSANDRA-10920, changes to /var/run/cassandra folder 
are not persistent across reboots and therefore saving pid file to that 
location fails. This causes trouble when managing the status of the cassandra 
service.

The linked issue mentions good enough (and working) workaround (compared to 
writing new service file for systemd). 

Trunk now contains files necessary to produce RHEL compatible RPM in the 
/redhat folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751349#comment-15751349
 ] 

Sylvain Lebresne commented on CASSANDRA-12620:
--

Thanks, +1 assuming the tests are clean (nit: I don't think you need the 
{{Row.Deletion}} import anymore).

> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> {code}
> {
> "partition" : {
>   "key" : [ "0", "26492" ],
>   "position" : 54397932
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 54397961,
> "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", 
> "773665449947099136" ],
> "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : 
> 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false },
> "cells" : [
>   { "name" : "identity_sid", "value" : "p_tw_zahidana" },
>   { "name" : "entities", "deletion_info" : { "marked_deleted" : 
> "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } 
> },
>   { "name" : "entities", "path" : [ 
> "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : 
> "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" },
>   { "name" : "entities", "path" : [ 
> "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" }
> ]
>   },
>   {
> "type" : "row",
>},
>   {
> "type" : "row",
> "position" : 54397177,
> "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", 
> "765850666296225792" ],
> "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397227,
> "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", 
> "765805367347601409" ],
> "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397276,
> "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", 
> "765760069858365441" ],
> "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" },
> "cells" : [ ]
>   },
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751327#comment-15751327
 ] 

Benjamin Lerer commented on CASSANDRA-12620:


Thanks for the review.
I updated my 
[branch|https://github.com/apache/cassandra/compare/trunk...blerer:12620-3.0?expand=1]
 with your change and tested it. It work perfectly including the usecase that 
was not working with my original fix.

I will run the upgrade tests and post the results once I got them.

> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> {code}
> {
> "partition" : {
>   "key" : [ "0", "26492" ],
>   "position" : 54397932
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 54397961,
> "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", 
> "773665449947099136" ],
> "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : 
> 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false },
> "cells" : [
>   { "name" : "identity_sid", "value" : "p_tw_zahidana" },
>   { "name" : "entities", "deletion_info" : { "marked_deleted" : 
> "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } 
> },
>   { "name" : "entities", "path" : [ 
> "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : 
> "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" },
>   { "name" : "entities", "path" : [ 
> "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" }
> ]
>   },
>   {
> "type" : "row",
>},
>   {
> "type" : "row",
> "position" : 54397177,
> "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", 
> "765850666296225792" ],
> "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397227,
> "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", 
> "765805367347601409" ],
> "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397276,
> "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", 
> "765760069858365441" ],
> "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" },
> "cells" : [ ]
>   },
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751089#comment-15751089
 ] 

Sylvain Lebresne edited comment on CASSANDRA-12620 at 12/15/16 12:47 PM:
-

Not sure this is the proper fix (as in, it's likely fine for anyone that isn't 
doing wacky things, but it doesn't stricly respect backward compatibility).

In 2.x, I believe we never delete a row marker explicitely, so it can only be a 
tombstone due to TTLs. But a row marker expiring does not absolutely guarantee 
the whole row is gone. Consider the following example (which is weird, for 
sure, but possible):
{noformat}
CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int);
UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0;
INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3;
{noformat}
When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no 
row marker internally), and this both in 2.x and 3.x. With this patch however, 
we might end up generating a row deletion with timestamp 1, which _would_ 
delete the first UPDATE, which is stricly speaking incorrect (for the CQL 
semantic).

Long story short, a deleted row marker is not equivalent to a row deletion, it 
just mean there is no row marker, and as the equivalent in 3.x of a row marker 
is the primary key liveness info, all we want here is make sure we don't have 
one. In other words, I believe the proper patch is:
{noformat}
assert !cell.value.hasRemaining();
-   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
+   if (!cell.isTombstone())
+   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
{noformat}



was (Author: slebresne):
Not sure this is the proper fix (as in, it's likely fine for anyone that isn't 
doing wacky things, but it doesn't stricly respect backward compatibility).

In 2.x, I believe we never delete a row marker explicitely, so it can only be a 
tombstone due to TTLs. But a row marker expiring does not absolutely guarantee 
the whole row is gone. Consider the following example (which is weird, for 
sure, but possible):
{noformat}
CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int);
UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0;
INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3;
{noformat}
When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no 
row marker internally), and this both in 2.x and 3.x. With this patch however, 
we might end up generating a row deletion with timestamp 1, which _would_ 
delete the first UPDATE, which is stricly speaking incorrect (for the CQL 
semantic).

Long story short, a deleted row marker is not equivalent to a row deletion, it 
just mean there is no row marker, and as the equivalent in 3.x of a row marker 
is the primary key liveness info, all we want here is make sure we don't have 
one. In other words, I believe the proper patch is:
{noformat}
assert !cell.value.hasRemaining();
-   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
+   if (cell.isTombstone())
+   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
{noformat}


> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> 

[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751089#comment-15751089
 ] 

Sylvain Lebresne commented on CASSANDRA-12620:
--

Not sure this is the proper fix (as in, it's likely fine for anyone that isn't 
doing wacky things, but it doesn't stricly respect backward compatibility).

In 2.x, I believe we never delete a row marker explicitely, so it can only be a 
tombstone due to TTLs. But a row marker expiring does not absolutely guarantee 
the whole row is gone. Consider the following example (which is weird, for 
sure, but possible):
{noformat}
CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int);
UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0;
INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3;
{noformat}
When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no 
row marker internally), and this both in 2.x and 3.x. With this patch however, 
we might end up generating a row deletion with timestamp 1, which _would_ 
delete the first UPDATE, which is stricly speaking incorrect (for the CQL 
semantic).

Long story short, a deleted row marker is not equivalent to a row deletion, it 
just mean there is no row marker, and as the equivalent in 3.x of a row marker 
is the primary key liveness info, all we want here is make sure we don't have 
one. In other words, I believe the proper patch is:
{noformat}
assert !cell.value.hasRemaining();
-   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
+   if (cell.isTombstone())
+   builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, 
cell.ttl, cell.localDeletionTime));
{noformat}


> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> {code}
> {
> "partition" : {
>   "key" : [ "0", "26492" ],
>   "position" : 54397932
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 54397961,
> "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", 
> "773665449947099136" ],
> "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : 
> 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false },
> "cells" : [
>   { "name" : "identity_sid", "value" : "p_tw_zahidana" },
>   { "name" : "entities", "deletion_info" : { "marked_deleted" : 
> "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } 
> },
>   { "name" : "entities", "path" : [ 
> "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : 
> "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" },
>   { "name" : "entities", "path" : [ 
> "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" }
> ]
>   },
>   {
> "type" : "row",
>},
>   {
> "type" : "row",
> "position" : 54397177,
> "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", 
> "765850666296225792" ],
> "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397227,
> 

[jira] [Updated] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-12620:
---
Reviewer: Sylvain Lebresne
  Status: Patch Available  (was: In Progress)

> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> {code}
> {
> "partition" : {
>   "key" : [ "0", "26492" ],
>   "position" : 54397932
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 54397961,
> "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", 
> "773665449947099136" ],
> "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : 
> 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false },
> "cells" : [
>   { "name" : "identity_sid", "value" : "p_tw_zahidana" },
>   { "name" : "entities", "deletion_info" : { "marked_deleted" : 
> "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } 
> },
>   { "name" : "entities", "path" : [ 
> "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : 
> "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" },
>   { "name" : "entities", "path" : [ 
> "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" }
> ]
>   },
>   {
> "type" : "row",
>},
>   {
> "type" : "row",
> "position" : 54397177,
> "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", 
> "765850666296225792" ],
> "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397227,
> "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", 
> "765805367347601409" ],
> "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397276,
> "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", 
> "765760069858365441" ],
> "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" },
> "cells" : [ ]
>   },
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751054#comment-15751054
 ] 

Benjamin Lerer commented on CASSANDRA-12620:


Sorry, for the delay. It tooks me some time and a certain amount of back and 
forth between the {{2.1}} and {{3.0}} code base to fully understand the problem.
[~bric3] Thanks for the SSTables they helped a lot.

In {{2.1}} when a row is created with a {{TTL}} the row marker cell is created 
as an {{ExpiringCell}} but when the row is compacted if the {{TTL}} has expired 
the cell will be converted into a {{DeletedCell}}.

In {{3.0}} the code used to read the post 3.0 format was only expecting an 
{{ExpiringCell}} for rows with {{TTL}} and was ignoring the local deletion time 
of the {{DeletedCell}}. Due to that the row was not marked as deleted anymore.

As the problem was in the code used to deserialize the post 3.0 format it was 
not necessary to run {{upgradesstables}} to see the problem.

I originally could not reproduce the problem because I did not think of running 
a compaction before upgrading (SizeTieredCompaction).

I pushed a patch 
[here|https://github.com/apache/cassandra/compare/trunk...blerer:12620-3.0?expand=1].

   



> Resurrected empty rows on update to 3.x
> ---
>
> Key: CASSANDRA-12620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Collin Sauve
>Assignee: Benjamin Lerer
>
> We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 
> according to documentation), and were entering TTLs at write-time using the 
> DataStax C# Driver (using the POCO mapper).
> Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that:
> * should have been TTL'd
> * have no non-primary-key column data
> {code}
> CREATE TABLE applicationservices.aggregate_bucket_event_v3 (
> bucket_type int,
> bucket_id text,
> date timestamp,
> aggregate_id text,
> event_type int,
> event_id text,
> entities list>>,
> identity_sid text,
> PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, 
> event_id)
> ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, 
> event_id ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> {code}
> {
> "partition" : {
>   "key" : [ "0", "26492" ],
>   "position" : 54397932
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 54397961,
> "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", 
> "773665449947099136" ],
> "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : 
> 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false },
> "cells" : [
>   { "name" : "identity_sid", "value" : "p_tw_zahidana" },
>   { "name" : "entities", "deletion_info" : { "marked_deleted" : 
> "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } 
> },
>   { "name" : "entities", "path" : [ 
> "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : 
> "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" },
>   { "name" : "entities", "path" : [ 
> "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" }
> ]
>   },
>   {
> "type" : "row",
>},
>   {
> "type" : "row",
> "position" : 54397177,
> "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", 
> "765850666296225792" ],
> "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397227,
> "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", 
> "765805367347601409" ],
> "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" },
> "cells" : [ ]
>   },
>   {
> "type" : "row",
> "position" : 54397276,
> "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", 
> "765760069858365441" ],
> "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" },
> "cells" : [ ]
>   },
> {code}



--
This message was 

[jira] [Commented] (CASSANDRA-12990) More fixes to the TokenAllocator

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751003#comment-15751003
 ] 

Branimir Lambov commented on CASSANDRA-12990:
-

Since we don't get the rack info, the new adapter should not [offer a means of 
getting 
it|https://github.com/DikangGu/cassandra/commit/d986e89b7b21b5464bd1de7908ee5cb1d12dfd30#diff-bf046c281d96767f4c3c4fe97eeaf938R220]
 at all. That method shouldn't be called by the no-replication allocator, let's 
make that clear there and then by adding a comment and throwing.

The adapter should also just return 1 for the replica count, so that the [0 
check in 
{{TokenAllocatorFactory}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/tokenallocator/TokenAllocatorFactory.java#L37]
 can be removed and the 0 replica hack is only done in one place in the code.

Now that {{waitForGossipToSettle}} is in {{Gossiper}}, the name is somewhat 
redundant. Rename to {{waitToSettle}}?

> More fixes to the TokenAllocator
> 
>
> Key: CASSANDRA-12990
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12990
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 3.12
>
>
> CASSANDRA-12983 fixes the NoReplicationTokenAllocator in the replication 
> factor equals 0 case.
> We find one more problem that, in our big cluster, ~1000 nodes, the Gossip 
> takes longer time to settle down, so the TokenAllocator may throw NPE when 
> trying to find the dc rack information from the Gossip metadata, in the 
> getStrategy function.
> This patch will fix it, it does two things:
> 1. wait gossip to settle down before trying to allocating tokens.
> 2. for replication factor equals 0 or 1 case, we do not need to check the 
> topology for rack information, since each node will be treated separately. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750958#comment-15750958
 ] 

Benjamin Roth edited comment on CASSANDRA-12905 at 12/15/16 10:00 AM:
--

+1 for the dtest!

Thanks for explaining that hint-thing.

For my understanding of hint retransmission:
If we have a hint file that is being processed and a WTE occurs in the middle, 
will the whole file be retransmitted or can it be resumed at the last 
successful position? 

I guess this is not the case from my personal observations. I had situations 
with > 1GB hint queues per sender node which were not going to shrink due to 
WTEs. It seemed like the same hints have been retransmitted over and over again 
from scratch instead of trying to resume. What helped in this situation was to 
pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs 
due to lock contentions. To be honest, I did not try lowering 
hinted_handoff_throttle_in_kb or at least I don't remember.


was (Author: brstgt):
+1 for the dtest!

For my understanding of hint retransmission:
If we have a hint file that is being processed and a WTE occurs in the middle, 
will the whole file be retransmitted or can it be resumed at the last 
successful position? 

I guess this is not the case from my personal observations. I had situations 
with > 1GB hint queues per sender node which were not going to shrink due to 
WTEs. It seemed like the same hints have been retransmitted over and over again 
from scratch instead of trying to resume. What helped in this situation was to 
pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs 
due to lock contentions. To be honest, I did not try lowering 
hinted_handoff_throttle_in_kb or at least I don't remember.

> Retry acquire MV lock on failure instead of throwing WTE on streaming
> -
>
> Key: CASSANDRA-12905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: centos 6.7 x86_64
>Reporter: Nir Zilka
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.10
>
>
> Hello,
> I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, 
> private VLAN),
> first it was 2.2.5.1 and repair worked flawlessly,
> second upgrade was to 3.0.9 (with upgradesstables) and also repair worked 
> well,
> then i upgraded 2 weeks ago to 3.9 - and the repair problems started.
> there are several errors types from the system.log (different nodes) :
> - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
> - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation 
> timed out - received only 0 responses
> - Remote peer xxx.xxx.xxx.xxx failed stream session
> - Session completed with the following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> 
> i use 3.9 default configuration with the cluster settings adjustments (3 
> seeds, GossipingPropertyFileSnitch).
> streaming_socket_timeout_in_ms is the default (8640).
> i'm afraid from consistency problems while i'm not performing repair.
> Any ideas?
> Thanks,
> Nir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750958#comment-15750958
 ] 

Benjamin Roth commented on CASSANDRA-12905:
---

+1 for the dtest!

For my understanding of hint retransmission:
If we have a hint file that is being processed and a WTE occurs in the middle, 
will the whole file be retransmitted or can it be resumed at the last 
successful position? 

I guess this is not the case from my personal observations. I had situations 
with > 1GB hint queues per sender node which were not going to shrink due to 
WTEs. It seemed like the same hints have been retransmitted over and over again 
from scratch instead of trying to resume. What helped in this situation was to 
pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs 
due to lock contentions. To be honest, I did not try lowering 
hinted_handoff_throttle_in_kb or at least I don't remember.

> Retry acquire MV lock on failure instead of throwing WTE on streaming
> -
>
> Key: CASSANDRA-12905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: centos 6.7 x86_64
>Reporter: Nir Zilka
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.10
>
>
> Hello,
> I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, 
> private VLAN),
> first it was 2.2.5.1 and repair worked flawlessly,
> second upgrade was to 3.0.9 (with upgradesstables) and also repair worked 
> well,
> then i upgraded 2 weeks ago to 3.9 - and the repair problems started.
> there are several errors types from the system.log (different nodes) :
> - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
> - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation 
> timed out - received only 0 responses
> - Remote peer xxx.xxx.xxx.xxx failed stream session
> - Session completed with the following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> 
> i use 3.9 default configuration with the cluster settings adjustments (3 
> seeds, GossipingPropertyFileSnitch).
> streaming_socket_timeout_in_ms is the default (8640).
> i'm afraid from consistency problems while i'm not performing repair.
> Any ideas?
> Thanks,
> Nir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908
 ] 

Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:43 AM:
-

I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
 * Calculates the likeliness of a race condition leading to unnecessary repairs
 * @see https://issues.apache.org/jira/browse/CASSANDRA-12991
 *
 * This assumes that all writes are equally spread over all token ranges
 * and there is one subrange repair executed for each existing token range

3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{quote}

You may ask why I entered latencies like 10ms or 20ms - this seems quite high. 
It is indeed quite high for regular tables and a cluster that is not 
overloaded. Under these conditions, the latency is dominated by your network 
latency, so 1ms seems quite fair to me.
As soon as you use MVs and your cluster tends to overload, higher latencies are 
not unrealistic.
You have to take into account that an MV operation does read before write and 
the latency may vary very much. For MVs the latency is not (only) any more 
dominated by network latency but by MV lock aquisition and read before write. 
Both factors can introduce MUCH higher latencies, depending on concurrent 
operations on MV, number of SSTables, compaction strategy, just everything that 
affects read performance.
If your cluster is overloaded, these effects have an even higher impact.

I observed MANY situations on our production system where writes timed out 
during streaming because of lock contention and or RBW impacts. These 
situations mainly pop up during repair sessions when streams cause bulk 
mutation applies (see StreamReceiverTask path for MVs). Impact is even higher 
due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the 
situation even more unpredictable and increases "drifts" of nodes, like Node A 
is overloaded but Node B not because Node A receives a stream from a different 
repair but Node B does not.

This is a vicious circle driven several factors: 
- Stream puts pressure on nodes - especially larg(er) partitions
- hints tend to queue up
- hint delivery puts more pressure
- retransmission of failed hint delivery puts even more pressure
- latencies go up
- stream validations drift
- more (unnecessary) streams
- goto 0

This calculation example is just hypothetic. This *may* happen as calculated 
but it totally depends on the model, cluster dimensions, cluster load, write 
activity, distribution of writes and repair execution. I don't claim that 
fixing this issue will remove all MV performance problems but it may be helps 
to remove one impediment in the mentioned vicious circle.

My proposal is NOT to control flushes. This is far too complicated and wont 
help. A flush, whenever it may happen and whatever range it flushes may or may 
not contain a mutation that _should_ be there. What helps is to cut off all 
data retrospectively at a synchronized and fix timestamp when executing the 
validation. You can define a grace period (GP). When you start validation at VS 
on the repair coordinator, then you expect all mutations to arrive no later 
than VS that were created before VS - GP. That can be done at SSTable scanner 
level by filtering all events (cells, tombstones) after VS - GP during 
validation compaction. Something like the opposite of purging tombstones after 
GCGS.


was (Author: brstgt):
I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
 * Calculates the likeliness of a race condition leading to unnecessary repairs
 * @see https://issues.apache.org/jira/browse/CASSANDRA-12991
 *
 * This assumes that all writes are equally spread over all token ranges
 * and there is one subrange repair executed for each existing token range

3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation 

[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908
 ] 

Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:34 AM:
-

I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
 * Calculates the likeliness of a race condition leading to unnecessary repairs
 * @see https://issues.apache.org/jira/browse/CASSANDRA-12991
 *
 * This assumes that all writes are equally spread over all token ranges
 * and there is one subrange repair executed for each existing token range

3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{quote}

You may ask why I entered latencies like 10ms or 20ms - this seems quite high. 
It is indeed quite high for regular tables and a cluster that is not 
overloaded. Under these conditions, the latency is dominated by your network 
latency, so 1ms seems quite fair to me.
As soon as you use MVs and your cluster tends to overload, higher latencies are 
not unrealistic.
You have to take into account that an MV operation does read before write and 
the latency may vary very much. For MVs the latency is not (only) any more 
dominated by network latency but by MV lock aquisition and read before write. 
Both factors can introduce MUCH higher latencies, depending on concurrent 
operations on MV, number of SSTables, compaction strategy, just everything that 
affects read performance.
If your cluster is overloaded, these effects have an even higher impact.

I observed MANY situations on our production system where writes timed out 
during streaming because of lock contention and or RBW impacts. These 
situations mainly pop up during repair sessions when streams cause bulk 
mutation applies (see StreamReceiverTask path for MVs). Impact is even higher 
due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the 
situation even more unpredictable and increases "drifts" of nodes, like Node A 
is overloaded but Node B not because Node A receives a stream from a different 
repair but Node B does not.

This is a vicious circle driven several factors: 
- Stream puts pressure on nodes - especially larg(er) partitions
- hints tend to queue up
- hint delivery puts more pressure
- retransmission of failed hint delivery puts even more pressure
- latencies go up
- stream validations drift
- more (unnecessary) streams
- goto 0

This calculation example is just hypothetic. This *may* happen as calculated 
but it totally depends on the model, cluster dimensions, cluster load, write 
activity, distribution of writes and repair execution. I don't claim that 
fixing this issue will remove all MV performance problems but it may be helps 
to remove one impediment in the mentioned vicious circle.

My proposal is NOT to control flushes. This is far too complicated and wont 
help. A flush, whenever it may happen and whatever range it flushes may or may 
not contain a mutation that _should_ be there. The only thing that helps is to 
cut off all data retrospectively at a synchronized and fix timestamp when 
executing the validation. You can only define a grace period (GP). When you 
start validation at VS on the repair coordinator, then you expect all mutations 
to arrive no later than VS that were created before VS - GP. That can IMHO only 
be done at SSTable scanner level by filtering all events (cells, tombstones) 
after VS - GP during validation compaction. Something like the opposite of 
purging tombstones after GCGS.


was (Author: brstgt):
I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for 

[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908
 ] 

Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:34 AM:
-

I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{quote}

You may ask why I entered latencies like 10ms or 20ms - this seems quite high. 
It is indeed quite high for regular tables and a cluster that is not 
overloaded. Under these conditions, the latency is dominated by your network 
latency, so 1ms seems quite fair to me.
As soon as you use MVs and your cluster tends to overload, higher latencies are 
not unrealistic.
You have to take into account that an MV operation does read before write and 
the latency may vary very much. For MVs the latency is not (only) any more 
dominated by network latency but by MV lock aquisition and read before write. 
Both factors can introduce MUCH higher latencies, depending on concurrent 
operations on MV, number of SSTables, compaction strategy, just everything that 
affects read performance.
If your cluster is overloaded, these effects have an even higher impact.

I observed MANY situations on our production system where writes timed out 
during streaming because of lock contention and or RBW impacts. These 
situations mainly pop up during repair sessions when streams cause bulk 
mutation applies (see StreamReceiverTask path for MVs). Impact is even higher 
due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the 
situation even more unpredictable and increases "drifts" of nodes, like Node A 
is overloaded but Node B not because Node A receives a stream from a different 
repair but Node B does not.

This is a vicious circle driven several factors: 
- Stream puts pressure on nodes - especially larg(er) partitions
- hints tend to queue up
- hint delivery puts more pressure
- retransmission of failed hint delivery puts even more pressure
- latencies go up
- stream validations drift
- more (unnecessary) streams
- goto 0

This calculation example is just hypothetic. This *may* happen as calculated 
but it totally depends on the model, cluster dimensions, cluster load, write 
activity, distribution of writes and repair execution. I don't claim that 
fixing this issue will remove all MV performance problems but it may be helps 
to remove one impediment in the mentioned vicious circle.

My proposal is NOT to control flushes. This is far too complicated and wont 
help. A flush, whenever it may happen and whatever range it flushes may or may 
not contain a mutation that _should_ be there. The only thing that helps is to 
cut off all data retrospectively at a synchronized and fix timestamp when 
executing the validation. You can only define a grace period (GP). When you 
start validation at VS on the repair coordinator, then you expect all mutations 
to arrive no later than VS that were created before VS - GP. That can IMHO only 
be done at SSTable scanner level by filtering all events (cells, tombstones) 
after VS - GP during validation compaction. Something like the opposite of 
purging tombstones after GCGS.


was (Author: brstgt):
I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{/quote}

You may ask why I entered 

[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908
 ] 

Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:33 AM:
-

I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{/quote}

You may ask why I entered latencies like 10ms or 20ms - this seems quite high. 
It is indeed quite high for regular tables and a cluster that is not 
overloaded. Under these conditions, the latency is dominated by your network 
latency, so 1ms seems quite fair to me.
As soon as you use MVs and your cluster tends to overload, higher latencies are 
not unrealistic.
You have to take into account that an MV operation does read before write and 
the latency may vary very much. For MVs the latency is not (only) any more 
dominated by network latency but by MV lock aquisition and read before write. 
Both factors can introduce MUCH higher latencies, depending on concurrent 
operations on MV, number of SSTables, compaction strategy, just everything that 
affects read performance.
If your cluster is overloaded, these effects have an even higher impact.

I observed MANY situations on our production system where writes timed out 
during streaming because of lock contention and or RBW impacts. These 
situations mainly pop up during repair sessions when streams cause bulk 
mutation applies (see StreamReceiverTask path for MVs). Impact is even higher 
due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the 
situation even more unpredictable and increases "drifts" of nodes, like Node A 
is overloaded but Node B not because Node A receives a stream from a different 
repair but Node B does not.

This is a vicious circle driven several factors: 
- Stream puts pressure on nodes - especially larg(er) partitions
- hints tend to queue up
- hint delivery puts more pressure
- retransmission of failed hint delivery puts even more pressure
- latencies go up
- stream validations drift
- more (unnecessary) streams
- goto 0

This calculation example is just hypothetic. This *may* happen as calculated 
but it totally depends on the model, cluster dimensions, cluster load, write 
activity, distribution of writes and repair execution. I don't claim that 
fixing this issue will remove all MV performance problems but it may be helps 
to remove one impediment in the mentioned vicious circle.

My proposal is NOT to control flushes. This is far too complicated and wont 
help. A flush, whenever it may happen and whatever range it flushes may or may 
not contain a mutation that _should_ be there. The only thing that helps is to 
cut off all data retrospectively at a synchronized and fix timestamp when 
executing the validation. You can only define a grace period (GP). When you 
start validation at VS on the repair coordinator, then you expect all mutations 
to arrive no later than VS that were created before VS - GP. That can IMHO only 
be done at SSTable scanner level by filtering all events (cells, tombstones) 
after VS - GP during validation compaction. Something like the opposite of 
purging tombstones after GCGS.


was (Author: brstgt):
I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
 * Calculates the likeliness of a race condition leading to unnecessary repairs
 * @see https://issues.apache.org/jira/browse/CASSANDRA-12991
 *
 * This assumes that all writes are equally spread over all token ranges
 * and there is one subrange repair executed for each existing token range

3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness 

[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908
 ] 

Benjamin Roth commented on CASSANDRA-12991:
---

I created a little script to calculate some possible scenarios: 
https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99

Output:
{quote}
 * Calculates the likeliness of a race condition leading to unnecessary repairs
 * @see https://issues.apache.org/jira/browse/CASSANDRA-12991
 *
 * This assumes that all writes are equally spread over all token ranges
 * and there is one subrange repair executed for each existing token range

3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 0.39%
Unnecessary range repairs per repair: 3.00

3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 
req/s
Total Ranges: 768
Likeliness for RC per range: 1.56%
Unnecessary range repairs per repair: 12.00

8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 2.93%
Unnecessary range repairs per repair: 60.00

8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 
req/s
Total Ranges: 2048
Likeliness for RC per range: 5.37%
Unnecessary range repairs per repair: 110.00
{/quote}

You may ask why I entered latencies like 10ms or 20ms - this seems quite high. 
It is indeed quite high for regular tables and a cluster that is not 
overloaded. Under these conditions, the latency is dominated by your network 
latency, so 1ms seems quite fair to me.
As soon as you use MVs and your cluster tends to overload, higher latencies are 
not unrealistic.
You have to take into account that an MV operation does read before write and 
the latency may vary very much. For MVs the latency is not (only) any more 
dominated by network latency but by MV lock aquisition and read before write. 
Both factors can introduce MUCH higher latencies, depending on concurrent 
operations on MV, number of SSTables, compaction strategy, just everything that 
affects read performance.
If your cluster is overloaded, these effects have an even higher impact.

I observed MANY situations on our production system where writes timed out 
during streaming because of lock contention and or RBW impacts. These 
situations mainly pop up during repair sessions when streams cause bulk 
mutation applies (see StreamReceiverTask path for MVs). Impact is even higher 
due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the 
situation even more unpredictable and increases "drifts" of nodes, like Node A 
is overloaded but Node B not because Node A receives a stream from a different 
repair but Node B does not.

This is a vicious circle driven several factors: 
- Stream puts pressure on nodes - especially larg(er) partitions
- hints tend to queue up
- hint delivery puts more pressure
- retransmission of failed hint delivery puts even more pressure
- latencies go up
- stream validations drift
- more (unnecessary) streams
- goto 0

This calculation example is just hypothetic. This *may* happen as calculated 
but it totally depends on the model, cluster dimensions, cluster load, write 
activity, distribution of writes and repair execution. I don't claim that 
fixing this issue will remove all MV performance problems but it may be helps 
to remove one impediment in the mentioned vicious circle.

My proposal is NOT to control flushes. This is far too complicated and wont 
help. A flush, whenever it may happen and whatever range it flushes may or may 
not contain a mutation that _should_ be there. The only thing that helps is to 
cut off all data retrospectively at a synchronized and fix timestamp when 
executing the validation. You can only define a grace period (GP). When you 
start validation at VS on the repair coordinator, then you expect all mutations 
to arrive no later than VS that were created before VS - GP. That can IMHO only 
be done at SSTable scanner level by filtering all events (cells, tombstones) 
after VS - GP during validation compaction. Something like the opposite of 
purging tombstones after GCGS.

> Inter-node race condition in validation compaction
> --
>
> Key: CASSANDRA-12991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12991
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Priority: Minor
>
> Problem:
> When a validation compaction is triggered by a repair it may happen that due 
> to flying in mutations the merkle trees differ but the data is consistent 
> however.
> Example:
> t = 1: 
> Repair starts, triggers validations
> Node A starts validation
> t = 10001:
> Mutation arrives at Node A
> t = 10002:
> Mutation arrives at 

[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error

[Nitsan Wakart (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750847#comment-15750847
 ] 

Nitsan Wakart commented on CASSANDRA-13006:
---

OnOutOfMemoryError and HeapDumpOnOutOfMemoryError are both available on Zing.
The default for HeapDumpOnOutOfMemoryError is false.
Let me know if you need more details.

> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by 
> default if the process encountered an OOM error. These heap dumps are stored 
> in the Apache Cassandra home directory unless configured otherwise (see 
> [Cassandra Support 
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
>  for this feature).
>  
> The creation and storage of heap dumps aides debugging and investigative 
> workflows, but is not be desirable for a production environment where these 
> heap dumps may occupy a large amount of disk space and require manual 
> intervention for cleanups. 
>  
> Managing heap dumps on out of memory errors and configuring the paths for 
> these heap dumps are available as JVM options in JVM. The current behavior 
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. 
>  
> A patch can be proposed here that would make the heap dump on OOM error honor 
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate 
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM 
> option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8496) Remove MemtablePostFlusher

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750807#comment-15750807
 ] 

Branimir Lambov commented on CASSANDRA-8496:


Bumping priority as we have seen production issues made worse by incorrect 
behaviour of forced flush and snapshot after a failed flush.

> Remove MemtablePostFlusher
> --
>
> Key: CASSANDRA-8496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Assignee: Branimir Lambov
>
> To improve clearing of the CL, prevent infinite growth, and ensure the prompt 
> completion of tasks waiting on flush in the case of transient errors, large 
> flushes or slow disks, in 2.1 we could eliminate the post flusher altogether. 
> Since we now enforce that Memtables track contiguous ranges, a relatively 
> small change would permit Memtables to know the exact minimum as well as the 
> currently known exact maximum. The CL could easily track the total dirty 
> range, knowing that it must be contiguous, by using an AtomicLong instead of 
> an AtomicInteger, and tracking both the min/max seen, not just the max. The 
> only slight complexity will come in for tracking the _clean_ range as this 
> can now be non-contiguous, if there are 3 memtable flushes covering the same 
> CL segment, and one of them completes later. To solve this we can use an 
> interval tree since these operations are infrequent, so the extra overhead is 
> nominal. Once the interval tree completely overlaps the dirty range, we mark 
> the entire dirty range clean.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8496) Remove MemtablePostFlusher

[Branimir Lambov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Branimir Lambov updated CASSANDRA-8496:
---
Priority: Major  (was: Minor)

> Remove MemtablePostFlusher
> --
>
> Key: CASSANDRA-8496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Assignee: Branimir Lambov
>
> To improve clearing of the CL, prevent infinite growth, and ensure the prompt 
> completion of tasks waiting on flush in the case of transient errors, large 
> flushes or slow disks, in 2.1 we could eliminate the post flusher altogether. 
> Since we now enforce that Memtables track contiguous ranges, a relatively 
> small change would permit Memtables to know the exact minimum as well as the 
> currently known exact maximum. The CL could easily track the total dirty 
> range, knowing that it must be contiguous, by using an AtomicLong instead of 
> an AtomicInteger, and tracking both the min/max seen, not just the max. The 
> only slight complexity will come in for tracking the _clean_ range as this 
> can now be non-contiguous, if there are 3 memtable flushes covering the same 
> CL segment, and one of them completes later. To solve this we can use an 
> interval tree since these operations are infrequent, so the extra overhead is 
> nominal. Once the interval tree completely overlaps the dirty range, we mark 
> the entire dirty range clean.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)