[jira] [Assigned] (CASSANDRA-10285) Compaction running indefinitely on system.hints

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson reassigned CASSANDRA-10285:
---

Assignee: Marcus Eriksson

> Compaction running indefinitely on system.hints 
> 
>
> Key: CASSANDRA-10285
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10285
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Marcus Eriksson
>
> During my hints storage benchmarks, I've experienced an issue using C* 2.2. 
> The hints was never replayed. After a while (more than 24H ...), I noticed 
> that there were still compactions running on system.hints and that some new 
> ones was triggered every 10-20 minutes.
> To reproduce, we create a cluster of 2 nodes. RF=2 and we generate hints by 
> shutting down the node2.
> {code}
> ccm create --install-dir=`pwd` -n 2 local && ccm start
> ccm node1 stress -- write n=50M -rate threads=300 -port jmx=7198 -errors 
> ignore -schema replication\(factor=2\)
> # wait 5-6 seconds to get the schema creation propagated on all nodes, then 
> in another window, stop node 2
> ccm node2 stop
> # wait the initial 50M writes are finished, bring back node2 up and write 
> another 50M keys.
> ccm node2 start
> ccm node1 stress -- write n=50M -rate threads=300 -port jmx=7198 -errors 
> ignore -schema replication\(factor=2\)
> # You should get the initial compaction finished after 15-20 minutes. You can 
> set the mb throughput to 0 to get that done faster. 
> # Monitor the node1. the hints will never be replayed and you should see 
> compactions happening indefinitely. 
> {code}
> //cc [~krummas] [~yukim]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10342) Read defragmentation can cause unnecessary repairs

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746981#comment-14746981
 ] 

Marcus Eriksson commented on CASSANDRA-10342:
-

Patch [here|https://github.com/krummas/cassandra/commits/marcuse/10342] which 
only defragments if we only read from unrepaired sstables.

But I wonder if we should keep doing defragmentation at all in 2.2+ where 
incremental repair is default - we should probably do a few benchmarks if the 
over-repair is worth it. [~mambocab] do you have cycles to run the benchmarks? 
I pushed a branch without the defragmentation 
[here|https://github.com/krummas/cassandra/commits/marcuse/10342-no_defrag] - 
we would need to run a mixed workload with a few incremental repairs thrown in 
and compare read latency and amount of data streamed with standard 2.1

> Read defragmentation can cause unnecessary repairs
> --
>
> Key: CASSANDRA-10342
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10342
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Olsson
>Priority: Minor
>
> After applying the fix from CASSANDRA-10299 to the cluster we started having 
> a problem of ~20k small sstables appearing for the table with static data 
> when running incremental repair.
> In the logs there were several messages about flushes for that table, one for 
> each repaired range. The flushed sstables were 0.000kb in size with < 100 ops 
> in each. When checking cfstats there were several writes to that table, even 
> though we were only reading from it and read repair did not repair anything.
> After digging around in the codebase I noticed that defragmentation of data 
> can occur while reading, depending on the query and some other conditions. 
> This causes the read data to be inserted again to have it in a more recent 
> sstable, which can be a problem if that data was repaired using incremental 
> repair. The defragmentation is done in 
> [CollationController.java|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/CollationController.java#L151].
> I guess this wasn't a problem with full repairs since I assume that the 
> digest should be the same even if you have two copies of the same data. But 
> with incremental repair this will most probably cause a mismatch between 
> nodes if that data already was repaired, since the other nodes probably won't 
> have that data in their unrepaired set.
> --
> I can add that the problems on our cluster was probably due to the fact that 
> CASSANDRA-10299 caused the same data to be streamed multiple times and ending 
> up in several sstables. One of the conditions for the defragmentation is that 
> the number of sstables read during a read request have to be more than the 
> minimum number of sstables needed for a compaction(> 4 in our case). So 
> normally I don't think this would cause ~20k sstables to appear, we probably 
> hit an extreme.
> One workaround for this is to use another compaction strategy than STCS(it 
> seems to be the only affected strategy, atleast in 2.1), but the solution 
> might be to either make defragmentation configurable per table or avoid 
> reinserting the data if any of the sstables involved in the read are repaired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Fix build

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 0afe4beeb -> af0678dd4


Fix build


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af0678dd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af0678dd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af0678dd

Branch: refs/heads/cassandra-3.0
Commit: af0678dd48389a05b0fffb89d21b4e2f9d91f9d6
Parents: 0afe4be
Author: Sylvain Lebresne 
Authored: Wed Sep 16 09:44:16 2015 +0200
Committer: Sylvain Lebresne 
Committed: Wed Sep 16 09:44:16 2015 +0200

--
 .../cassandra/hadoop/cql3/CqlInputFormat.java   | 23 +++-
 1 file changed, 17 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/af0678dd/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java 
b/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
index 7699f29..3f74c33 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
@@ -19,19 +19,30 @@ package org.apache.cassandra.hadoop.cql3;
 
 import java.io.IOException;
 import java.util.*;
-import java.util.concurrent.*
+import java.util.concurrent.*;
+
+import com.datastax.driver.core.Host;
+import com.datastax.driver.core.Metadata;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Row;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.TokenRange;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import com.datastax.driver.core.*;
 import org.apache.cassandra.db.SystemKeyspace;
 import org.apache.cassandra.dht.*;
+import org.apache.cassandra.thrift.KeyRange;
 import org.apache.cassandra.hadoop.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.mapred.*;
-import org.apache.hadoop.mapreduce.*;
 
-import com.datastax.driver.core.Row;
 
 /**
  * Hadoop InputFormat allowing map/reduce against Cassandra rows within one 
ColumnFamily.

[jira] [Updated] (CASSANDRA-9839) Move crc_check_chance out of compressions options

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-9839:
-
Reviewer: Aleksey Yeschenko  (was: Benjamin Lerer)

> Move crc_check_chance out of compressions options
> -
>
> Key: CASSANDRA-9839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9839
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: client-impacting, docs-impacting
> Fix For: 3.0.0 rc1
>
>
> Follow up to CASSANDRA-8384. The option doesn't belong to compression params 
> - it doesn't affect compression, itself, and isn't passed to compressors upon 
> initialization.
> While it's true that it is (currently) only being honored when reading 
> compressed sstables, it still doesn't belong to compression params (and is 
> causing CASSANDRA-7978 -like issues).
> [~tjake] suggested we should make it an option of its own, and I think we 
> should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10345) incremental_repair_test.TestIncRepair.sstable_marking_test fails on cassandra-3.0

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747101#comment-14747101
 ] 

Sylvain Lebresne commented on CASSANDRA-10345:
--

Fyi, I've merged the dtest PR.

> incremental_repair_test.TestIncRepair.sstable_marking_test fails on 
> cassandra-3.0
> -
>
> Key: CASSANDRA-10345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10345
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Minor
> Fix For: 3.0.x
>
>
> incremental_repair_test.TestIncRepair.sstable_marking_test is failing often 
> on cassandra-3.0.
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/incremental_repair_test/TestIncRepair/sstable_marking_test/history/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10322) skipBytes is used extensively, but is slow

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747127#comment-14747127
 ] 

Benedict commented on CASSANDRA-10322:
--

Certainly I'm not wed to the current approach, but we skipBytes _a lot_, and we 
have to have a loop somewhere - having the loop inside of the virtual 
invocation makes the most sense. What I've effectively done here is strengthen 
the contract of {{skipBytes}} in {{DataInputPlus}} - but I haven't declared 
this in the interface (which at the time I was a little torn on - given every 
single {{DataInput}} we've used has always honoured this stronger contract), so 
I would like to suggest either:

# Redeclaring {{skipBytes}} in {{DataInputPlus}} and making it clear that any 
implementor must skip fully unless EOF is reached _only_
# Declaring a new {{skipBytesFully}} in {{DataInputPlus}}

I prefer the former, as it is less code; I just opted not to actually redeclare 
it.

> skipBytes is used extensively, but is slow
> --
>
> Key: CASSANDRA-10322
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10322
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.x
>
>
> We skip a great deal to avoid materializing data. Ironically, however, 
> skipping is just as (perhaps more) expensive, as it allocates a temporary 
> array of the size of the number of bytes we want to skip.
> This trivial patch implements {{skipBytes}} more efficiently, and simplifies 
> {{FileUtils.skipBytesFully}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/6] cassandra git commit: Merge branch cassandra-2.1 into cassandra-2.2

[blerer]

Merge branch cassandra-2.1 into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e7c2952d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e7c2952d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e7c2952d

Branch: refs/heads/trunk
Commit: e7c2952d31e8b6ddfef220942ba8306beabb
Parents: 8cad732 b8b4eb7
Author: blerer 
Authored: Wed Sep 16 10:21:44 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:21:58 2015 +0200

--
 build.xml|  1 +
 doc/cql3/CQL.textile | 20 ++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/build.xml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/doc/cql3/CQL.textile
--

[jira] [Commented] (CASSANDRA-9964) Document post-2.1 caching table options syntax

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747146#comment-14747146
 ] 

Benjamin Lerer commented on CASSANDRA-9964:
---

+1

> Document post-2.1 caching table options syntax
> --
>
> Key: CASSANDRA-9964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9964
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Paulo Motta
>Priority: Minor
> Fix For: 2.1.x, 2.2.x, 3.0.0 rc1
>
>
> CASSANDRA-5357 added new syntax for caching for {{CREATE TABLE}} in 2.1, but 
> [documentation|https://github.com/apache/cassandra/blob/trunk/doc/cql3/CQL.textile]
>  hasn't been updated accordingly.
> Now that the old options are gone completely after CASSANDRA-9712, we *must* 
> update the CQL spec.
> [DS 
> Docs|http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_set_caching_t.html]
>  have been up to date for a while.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10349) Rename DataOutputBuffer.getFilePointer() to position()

[Benedict (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benedict updated CASSANDRA-10349:
-
Summary: Rename DataOutputBuffer.getFilePointer() to position()  (was: 
Rename DataOutputBuffer.getFilePointer() to getPosition())

> Rename DataOutputBuffer.getFilePointer() to position()
> --
>
> Key: CASSANDRA-10349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10349
> Project: Cassandra
>  Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> {{DataOutputBuffer}} now supports {{getFilePointer()}} method - but the 
> nomenclature is actually wrong as it doesn't need to be a _file_ pointer. 
> Linked branch renames the two affected methods to {{getPosition()}} and 
> {{hasPosition()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7392) Abort in-progress queries that time out

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747201#comment-14747201
 ] 

Stefania commented on CASSANDRA-7392:
-

I've added the no spam logger and moved the logging to DEBUG, increasing the 
default max number of queries that we report to 50. I know aggregation doesn't 
make much sense but it won't hurt, in case an app sends the same query multiple 
times. 

I also prefer to re-introduce the CAS in {{MonitoringStateRef}}: if the worker 
thread does not notice that the query was aborted it will carry on iterating 
which defeats the purpose of aborting the queries. 

bq. They are usually going to be unique so there is nothing to aggregate on.

It's also worse than this. I didn't realize that the CQL string reconstructed 
from {{ReadCommand}} is really an approximation as we don't have all the 
information there. For example, a query without a condition on the primary key 
will be split in several queries as follows:

{code}
SELECT * FROM ks.test2 WHERE token(id) > -1976574744135038542 AND token(id) <= 
-1551387922747101229 LIMIT 5000: total time 10011 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 1096463829018333632 AND token(id) <= 
1355062136393692257 LIMIT 5000: total time 10078 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 8977444122753183931 AND token(id) <= 
9033798691964141178 LIMIT 5000: total time 10057 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 2635684107471725435 AND token(id) <= 
271655031657904 LIMIT 5000: total time 10078 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 9080285075713538993 AND token(id) <= 
9187108821678730728 LIMIT 5000: total time 10056 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > -8240319968209337270 AND token(id) <= 
-7817157413941317374 LIMIT 5000: total time 10032 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 8340735344052968255 AND token(id) <= 
8546322458038003371 LIMIT 5000: total time 10057 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 5722969564085623706 AND token(id) <= 
5806785306771146835 LIMIT 5000: total time 10073 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 7726207511295901422 AND token(id) <= 
7839180972141923302 LIMIT 5000: total time 10058 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 3532380910529882202 AND token(id) <= 
3654921169010564232 LIMIT 5000: total time 10074 msec - timeout 1 msec
SELECT * FROM ks.test2 WHERE token(id) > 7881865912870825334 AND token(id) <= 
7931494104861828509 LIMIT 5000: total time 10058 msec - timeout 1 msec
{code}

A query with a condition only on non primary keys and {{ALLOW FILTERING}] will 
not be reported as such since the filtering is not done on the worker. There 
may be other limitations.

To fix this we would either need to pass the original user query all the way to 
{{ReadCommand}}, which involves changing the serialization format and increases 
the memory footprint, or we need to move the reporting to the coordinator, or 
we would have to rely on a table like we do for tracing. Unless we try and 
squeeze the first option in before 3.0 hits, I think the other two options are 
best dealt with in a separate ticket where we focus more on logging rather than 
aborting queries.



> Abort in-progress queries that time out
> ---
>
> Key: CASSANDRA-7392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7392
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 3.x
>
>
> Currently we drop queries that time out before we get to them (because node 
> is overloaded) but not queries that time out while being processed.  
> (Particularly common for index queries on data that shouldn't be indexed.)  
> Adding the latter and logging when we have to interrupt one gets us a poor 
> man's "slow query log" for free.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747178#comment-14747178
 ] 

Benedict commented on CASSANDRA-10286:
--

[~ptnapoleon] this job failed completely, for what looks like environmental 
reasons (missing the tmp dirs it tried to create). Could you look into it, to 
see what the problem is?

We need to commit this before Friday; if we cannot get CI working, we'll have 
to commit it tomorrow and let the normal windows CI check it out.

> Windows utest 3.0: 
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError: 
>   at 
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete 
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException: 
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
>   at 
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
>  ~[na:1.8.0_51]
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>  ~[na:1.8.0_51]
>   at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
>  Source) [main/:na]
>   at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
>  Source) [main/:na]
>   at 
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) 
> [na:1.8.0_51]
>   at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540) 
> [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) 
> [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
>  [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) 
> [na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
>  [main/:na]
>   at 
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294) 
> [main/:na]
>   at 
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:172) 
> [main/:na]
>   at 
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:92) 
> [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.complete(TransactionLog.java:950)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.doAbort(TransactionLog.java:969)
>  

[jira] [Commented] (CASSANDRA-10267) Failing tests in upgrade_trests.paging_test

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747036#comment-14747036
 ] 

Benjamin Lerer commented on CASSANDRA-10267:


{quote}
{{upgrade_tests/paging_test.py:TestPagingWithDeletions.test_multiple_row_deletions}}
 is fixed by {{CASSANDRA-6237}}, but only for 3.0<->3.0, versions under 3.0 
don't appear to be targeted. Benjamin Lerer, can that be removed from the 
upgrade tests, or should it be working with upgrades from 2.1 to 3.0?
{quote}

The test can be removed. We were not supporting this type of deletion before 
3.0.

> Failing tests in upgrade_trests.paging_test
> ---
>
> Key: CASSANDRA-10267
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10267
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Sylvain Lebresne
>Assignee: Blake Eggleston
> Fix For: 3.0.0 rc1
>
>
> This is a continuation of CASSANDRA-9893 to deal with the failure of the 
> {{upgrade_trests.paging_test}} tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10050) Secondary Index Performance Dependent on TokenRange Searched in Analytics

[Sam Tunnicliffe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747104#comment-14747104
 ] 

Sam Tunnicliffe commented on CASSANDRA-10050:
-

bq.any thoughts on this? 
yes, but no bandwidth to do anything with them before 3.0

tl;dr  index entries are stored in key order, which is different from token 
order so when paging using tokens, the index lookup can't efficiently slice 
into the index table but has to scan and filter. I suspect this is related

> Secondary Index Performance Dependent on TokenRange Searched in Analytics
> -
>
> Key: CASSANDRA-10050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10050
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Single node, macbook, 2.1.8
>Reporter: Russell Alexander Spitzer
> Fix For: 3.x
>
>
> In doing some test work on the Spark Cassandra Connector I saw some odd 
> performance when pushing down range queries with Secondary Index filters. 
> When running the queries we see huge amount of time when the C* server is not 
> doing any work and the query seem to be hanging. This investigation led to 
> the work in this document
> https://docs.google.com/spreadsheets/d/1aJg3KX7nPnY77RJ9ZT-IfaYADgJh0A--nAxItvC6hb4/edit#gid=0
> The Spark Cassandra Connector builds up token range specific queries and 
> allows the user to pushdown relevant fields to C*. Here we have two indexed 
> fields (size) and (color) being pushed down to C*. 
> {code}
> SELECT count(*) FROM ks.tab WHERE token("store") > $min AND token("store") <= 
> $max AND color = 'red' AND size = 'P' ALLOW FILTERING;{code}
> These queries will have different token ranges inserted and executed as 
> separate spark tasks. Spark tasks with token ranges near the Min(token) end 
> up executing much faster than those near Max(token) which also happen to 
> through errors.
> {code}
> Coordinator node timed out waiting for replica nodes' responses] 
> message="Operation timed out - received only 0 responses." 
> info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
> {code}
> I took the queries and ran them through CQLSH to see the difference in time. 
> A linear relationship is seen based on where the tokenRange being queried is 
> starting with only 2 second for queries near the beginning of the full token 
> spectrum and over 12 seconds at the end of the spectrum. 
> The question is, can this behavior be improved? or should we not recommend 
> using secondary indexes with Analytics workloads?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] cassandra git commit: Document post-2.1 caching table options syntax

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 2cd59b875 -> b8b4eb76c


Document post-2.1 caching table options syntax

patch by Paulo Motta; reviewed by Benjamin Lerer for CASSANDRA-9964


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3b46e5bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3b46e5bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3b46e5bd

Branch: refs/heads/cassandra-2.1
Commit: 3b46e5bda61c0944ec28d56c289954d9f34c418e
Parents: 2cd59b8
Author: Paulo Motta 
Authored: Wed Sep 16 10:11:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:11:47 2015 +0200

--
 doc/cql3/CQL.textile | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3b46e5bd/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index a29ee6c..648c39f 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -327,13 +327,12 @@ Table creation supports the following other @@:
 |@dclocal_read_repair_chance@ | _simple_ | 0   | The probability with 
which to query extra nodes (e.g. more nodes than required by the consistency 
level) belonging to the same data center than the read coordinator for the 
purpose of read repairs.|
 |@gc_grace_seconds@   | _simple_ | 864000  | Time to wait before 
garbage collecting tombstones (deletion markers).|
 |@bloom_filter_fp_chance@ | _simple_ | 0.00075 | The target 
probability of false positive of the sstable bloom filters. Said bloom filters 
will be sized to provide the provided probability (thus lowering this value 
impact the size of bloom filters in-memory and on-disk)|
-|@compaction@ | _map_| _see below_ | The compaction 
options to use, see below.|
-|@compression@| _map_| _see below_ | Compression options, 
see below. |
-|@caching@| _simple_ | keys_only   | Whether to cache keys 
("key cache") and/or rows ("row cache") for this table. Valid values are: 
@all@, @keys_only@, @rows_only@ and @none@. |
 |@default_time_to_live@   | _simple_ | 0   | The default 
expiration time ("TTL") in seconds for a table.|
+|@compaction@ | _map_| _see below_ | Compaction options, 
see "below":#compactionOptions.|
+|@compression@| _map_| _see below_ | Compression options, 
see "below":#compressionOptions.|
+|@caching@| _map_| _see below_ | Caching options, see 
"below":#cachingOptions.|
 
-
-h4(#compactionOptions). @compaction@ options
+h4(#compactionOptions). Compaction options
 
 The @compaction@ property must at least define the @'class'@ sub-option, that 
defines the compaction strategy class to use. The default supported class are 
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@ and 
@'DateTieredCompactionStrategy'@. Custom strategy can be provided by specifying 
the full class name as a "string constant":#constants. The rest of the 
sub-options depends on the chosen class. The sub-options supported by the 
default classes are:
 
@@ -353,13 +352,22 @@ The @compaction@ property must at least define the 
@'class'@ sub-option, that de
 | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365 
 | SSTables only containing data that is older than this will never be 
compacted. |
 
 
-For the @compression@ property, the following default sub-options are 
available:
+h4(#compressionOptions). Compression options
+
+For the @compression@ property, the following sub-options are available:
 
 |_. option  |_. default|_. description |
 | @sstable_compression@ | LZ4Compressor| The compression algorithm to use. 
Default compressor are: LZ4Compressor, SnappyCompressor and DeflateCompressor. 
Use an empty string (@''@) to disable compression. Custom compressor can be 
provided by specifying the full class name as a "string constant":#constants.|
 | @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 | @crc_check_chance@| 1.0  | When compression is enabled, each 
compressed block includes a checksum of that block for the purpose of detecting 
disk bitrot and avoiding the propagation of corruption to other replica. This 
option defines the probability with which those checksums are checked during 
read. By default they are always checked. Set to 0 to disable 

[2/2] cassandra git commit: Add Benjamin Lerer to the developer list in build.xml

[blerer]

Add Benjamin Lerer to the developer list in build.xml

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8b4eb76
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8b4eb76
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8b4eb76

Branch: refs/heads/cassandra-2.1
Commit: b8b4eb76c42a61b4f33aa57041dac5af2ba2480c
Parents: 3b46e5b
Author: blerer 
Authored: Wed Sep 16 10:18:18 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:18:18 2015 +0200

--
 build.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8b4eb76/build.xml
--
diff --git a/build.xml b/build.xml
index 20c26a8..c36c819 100644
--- a/build.xml
+++ b/build.xml
@@ -415,6 +415,7 @@
 
 
 
+
 
 
 

[1/4] cassandra git commit: Document post-2.1 caching table options syntax

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 af0678dd4 -> 0d4e0cd22


Document post-2.1 caching table options syntax

patch by Paulo Motta; reviewed by Benjamin Lerer for CASSANDRA-9964


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3b46e5bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3b46e5bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3b46e5bd

Branch: refs/heads/cassandra-3.0
Commit: 3b46e5bda61c0944ec28d56c289954d9f34c418e
Parents: 2cd59b8
Author: Paulo Motta 
Authored: Wed Sep 16 10:11:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:11:47 2015 +0200

--
 doc/cql3/CQL.textile | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3b46e5bd/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index a29ee6c..648c39f 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -327,13 +327,12 @@ Table creation supports the following other @@:
 |@dclocal_read_repair_chance@ | _simple_ | 0   | The probability with 
which to query extra nodes (e.g. more nodes than required by the consistency 
level) belonging to the same data center than the read coordinator for the 
purpose of read repairs.|
 |@gc_grace_seconds@   | _simple_ | 864000  | Time to wait before 
garbage collecting tombstones (deletion markers).|
 |@bloom_filter_fp_chance@ | _simple_ | 0.00075 | The target 
probability of false positive of the sstable bloom filters. Said bloom filters 
will be sized to provide the provided probability (thus lowering this value 
impact the size of bloom filters in-memory and on-disk)|
-|@compaction@ | _map_| _see below_ | The compaction 
options to use, see below.|
-|@compression@| _map_| _see below_ | Compression options, 
see below. |
-|@caching@| _simple_ | keys_only   | Whether to cache keys 
("key cache") and/or rows ("row cache") for this table. Valid values are: 
@all@, @keys_only@, @rows_only@ and @none@. |
 |@default_time_to_live@   | _simple_ | 0   | The default 
expiration time ("TTL") in seconds for a table.|
+|@compaction@ | _map_| _see below_ | Compaction options, 
see "below":#compactionOptions.|
+|@compression@| _map_| _see below_ | Compression options, 
see "below":#compressionOptions.|
+|@caching@| _map_| _see below_ | Caching options, see 
"below":#cachingOptions.|
 
-
-h4(#compactionOptions). @compaction@ options
+h4(#compactionOptions). Compaction options
 
 The @compaction@ property must at least define the @'class'@ sub-option, that 
defines the compaction strategy class to use. The default supported class are 
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@ and 
@'DateTieredCompactionStrategy'@. Custom strategy can be provided by specifying 
the full class name as a "string constant":#constants. The rest of the 
sub-options depends on the chosen class. The sub-options supported by the 
default classes are:
 
@@ -353,13 +352,22 @@ The @compaction@ property must at least define the 
@'class'@ sub-option, that de
 | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365 
 | SSTables only containing data that is older than this will never be 
compacted. |
 
 
-For the @compression@ property, the following default sub-options are 
available:
+h4(#compressionOptions). Compression options
+
+For the @compression@ property, the following sub-options are available:
 
 |_. option  |_. default|_. description |
 | @sstable_compression@ | LZ4Compressor| The compression algorithm to use. 
Default compressor are: LZ4Compressor, SnappyCompressor and DeflateCompressor. 
Use an empty string (@''@) to disable compression. Custom compressor can be 
provided by specifying the full class name as a "string constant":#constants.|
 | @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 | @crc_check_chance@| 1.0  | When compression is enabled, each 
compressed block includes a checksum of that block for the purpose of detecting 
disk bitrot and avoiding the propagation of corruption to other replica. This 
option defines the probability with which those checksums are checked during 
read. By default they are always checked. Set to 0 to disable 

[2/4] cassandra git commit: Add Benjamin Lerer to the developer list in build.xml

[blerer]

Add Benjamin Lerer to the developer list in build.xml

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8b4eb76
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8b4eb76
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8b4eb76

Branch: refs/heads/cassandra-3.0
Commit: b8b4eb76c42a61b4f33aa57041dac5af2ba2480c
Parents: 3b46e5b
Author: blerer 
Authored: Wed Sep 16 10:18:18 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:18:18 2015 +0200

--
 build.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8b4eb76/build.xml
--
diff --git a/build.xml b/build.xml
index 20c26a8..c36c819 100644
--- a/build.xml
+++ b/build.xml
@@ -415,6 +415,7 @@
 
 
 
+
 
 
 

[3/4] cassandra git commit: Merge branch cassandra-2.1 into cassandra-2.2

[blerer]

Merge branch cassandra-2.1 into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e7c2952d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e7c2952d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e7c2952d

Branch: refs/heads/cassandra-3.0
Commit: e7c2952d31e8b6ddfef220942ba8306beabb
Parents: 8cad732 b8b4eb7
Author: blerer 
Authored: Wed Sep 16 10:21:44 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:21:58 2015 +0200

--
 build.xml|  1 +
 doc/cql3/CQL.textile | 20 ++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/build.xml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/doc/cql3/CQL.textile
--

[4/4] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d4e0cd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d4e0cd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d4e0cd2

Branch: refs/heads/cassandra-3.0
Commit: 0d4e0cd22653ff7c3c1ef4a4b2c23c78524ecc42
Parents: af0678d e7c2952
Author: blerer 
Authored: Wed Sep 16 10:33:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:34:00 2015 +0200

--
 build.xml|  1 +
 doc/cql3/CQL.textile | 20 ++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d4e0cd2/build.xml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d4e0cd2/doc/cql3/CQL.textile
--
diff --cc doc/cql3/CQL.textile
index 0e04528,ce712be..60e3d5e
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@@ -362,14 -361,22 +361,23 @@@ The @compaction@ property must at leas
  | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365
  | SSTables only containing data that is older than this will never be 
compacted. |
  
  
- For the @compression@ property, the following default sub-options are 
available:
+ h4(#compressionOptions). Compression options
+ 
+ For the @compression@ property, the following sub-options are available:
  
 -|_. option  |_. default|_. description |
 -| @sstable_compression@ | LZ4Compressor| The compression algorithm to 
use. Default compressor are: LZ4Compressor, SnappyCompressor and 
DeflateCompressor. Use an empty string (@''@) to disable compression. Custom 
compressor can be provided by specifying the full class name as a "string 
constant":#constants.|
 -| @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 -| @crc_check_chance@| 1.0  | When compression is enabled, 
each compressed block includes a checksum of that block for the purpose of 
detecting disk bitrot and avoiding the propagation of corruption to other 
replica. This option defines the probability with which those checksums are 
checked during read. By default they are always checked. Set to 0 to disable 
checksum checking and to 0.5 for instance to check them every other read|
 +|_. option |_. default|_. description |
 +| @class@  | LZ4Compressor| The compression algorithm to 
use. Default compressor are: LZ4Compressor, SnappyCompressor and 
DeflateCompressor. Use @'enabled' : false@ to disable compression. Custom 
compressor can be provided by specifying the full class name as a "string 
constant":#constants.|
 +| @enabled@| true | By default compression is 
enabled. To disable it, set @enabled@ to @false@
 +| @chunk_length_in_kb@ | 64KB | On disk SSTables are 
compressed by block (to allow random reads). This defines the size (in KB) of 
said block. Bigger values may improve the compression rate, but increases the 
minimum size of data to be read from disk for a read |
 +| @crc_check_chance@   | 1.0  | When compression is enabled, 
each compressed block includes a checksum of that block for the purpose of 
detecting disk bitrot and avoiding the propagation of corruption to other 
replica. This option defines the probability with which those checksums are 
checked during read. By default they are always checked. Set to 0 to disable 
checksum checking and to 0.5 for instance to check them every other read|
  
+ h4(#cachingOptions). Caching options
+ 
+ For the @caching@ property, the following sub-options are available:
+ 
+ |_. option  |_. default|_. description |
+ |@keys@ | ALL   | Whether to cache keys ("key cache") for 
this table. Valid values are: @ALL@ and @NONE@.|
+ |@rows_per_partition@   | NONE   | The amount of rows to cache per partition 
("row cache"). If an integer @n@ is specified, the first @n@ queried rows of a 
partition will be cached. Other possible options are @ALL@, to cache all rows 
of a queried partition, or @NONE@ to disable row caching.|
  
  h4. Other considerations:
  

[2/3] cassandra git commit: Correctly identify partition deletions for index transactions

[samt]

Correctly identify partition deletions for index transactions

Patch by Sam Tunnicliffe; reviewed by Sylvain Lebresne for
CASSANDRA-10343


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c694cd62
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c694cd62
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c694cd62

Branch: refs/heads/trunk
Commit: c694cd62af6f404ea0900e6e15410cbb8681841f
Parents: 0d4e0cd
Author: Sam Tunnicliffe 
Authored: Tue Sep 15 16:01:36 2015 +0100
Committer: Sam Tunnicliffe 
Committed: Wed Sep 16 09:37:15 2015 +0100

--
 .../db/partitions/AtomicBTreePartition.java |  2 +-
 .../org/apache/cassandra/index/StubIndex.java   | 14 ++
 .../index/internal/CustomIndexTest.java | 29 
 3 files changed, 38 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java 
b/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
index ae8a1c3..2be882e 100644
--- a/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
@@ -127,7 +127,7 @@ public class AtomicBTreePartition extends 
AbstractBTreePartition
 updater.ref = current;
 updater.reset();
 
-if (!update.deletionInfo().isLive())
+if (!update.deletionInfo().getPartitionDeletion().isLive())
 
indexer.onPartitionDeletion(update.deletionInfo().getPartitionDeletion());
 
 if (update.deletionInfo().hasRanges())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/test/unit/org/apache/cassandra/index/StubIndex.java
--
diff --git a/test/unit/org/apache/cassandra/index/StubIndex.java 
b/test/unit/org/apache/cassandra/index/StubIndex.java
index 0d3cf47..0ea03dc 100644
--- a/test/unit/org/apache/cassandra/index/StubIndex.java
+++ b/test/unit/org/apache/cassandra/index/StubIndex.java
@@ -38,6 +38,8 @@ import org.apache.cassandra.utils.concurrent.OpOrder;
 
 public class StubIndex implements Index
 {
+public List partitionDeletions = new ArrayList<>();
+public List rangeTombstones = new ArrayList<>();
 public List rowsInserted = new ArrayList<>();
 public List rowsDeleted = new ArrayList<>();
 public List> rowsUpdated = new ArrayList<>();
@@ -48,6 +50,8 @@ public class StubIndex implements Index
 {
 rowsInserted.clear();
 rowsDeleted.clear();
+partitionDeletions.clear();
+rangeTombstones.clear();
 }
 
 public StubIndex(ColumnFamilyStore baseCfs, IndexMetadata metadata)
@@ -58,11 +62,7 @@ public class StubIndex implements Index
 
 public boolean indexes(PartitionColumns columns)
 {
-for (ColumnDefinition col : columns)
-for (ColumnIdentifier indexed : indexMetadata.columns)
-if (indexed.equals(col.name))
-return true;
-return false;
+return true;
 }
 
 public boolean shouldBuildBlocking()
@@ -80,7 +80,7 @@ public class StubIndex implements Index
 return filter;
 }
 
-public Indexer indexerFor(DecoratedKey key,
+public Indexer indexerFor(final DecoratedKey key,
   int nowInSec,
   OpOrder.Group opGroup,
   IndexTransaction.Type transactionType)
@@ -93,10 +93,12 @@ public class StubIndex implements Index
 
 public void partitionDelete(DeletionTime deletionTime)
 {
+partitionDeletions.add(deletionTime);
 }
 
 public void rangeTombstone(RangeTombstone tombstone)
 {
+rangeTombstones.add(tombstone);
 }
 
 public void insertRow(Row row)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
--
diff --git a/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java 
b/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
index 255af4b..2f7a10b 100644
--- a/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
+++ b/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
@@ -57,6 +57,35 @@ public class CustomIndexTest extends CQLTester
 

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[samt]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48da81dc
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48da81dc
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48da81dc

Branch: refs/heads/trunk
Commit: 48da81dc4b64e2a064016458ac924036d8fe21c4
Parents: 61ce1b9 c694cd6
Author: Sam Tunnicliffe 
Authored: Wed Sep 16 09:40:14 2015 +0100
Committer: Sam Tunnicliffe 
Committed: Wed Sep 16 09:40:14 2015 +0100

--
 .../db/partitions/AtomicBTreePartition.java |  2 +-
 .../org/apache/cassandra/index/StubIndex.java   | 14 ++
 .../index/internal/CustomIndexTest.java | 29 
 3 files changed, 38 insertions(+), 7 deletions(-)
--

[1/3] cassandra git commit: Correctly identify partition deletions for index transactions

[samt]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 0d4e0cd22 -> c694cd62a
  refs/heads/trunk 61ce1b9a7 -> 48da81dc4


Correctly identify partition deletions for index transactions

Patch by Sam Tunnicliffe; reviewed by Sylvain Lebresne for
CASSANDRA-10343


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c694cd62
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c694cd62
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c694cd62

Branch: refs/heads/cassandra-3.0
Commit: c694cd62af6f404ea0900e6e15410cbb8681841f
Parents: 0d4e0cd
Author: Sam Tunnicliffe 
Authored: Tue Sep 15 16:01:36 2015 +0100
Committer: Sam Tunnicliffe 
Committed: Wed Sep 16 09:37:15 2015 +0100

--
 .../db/partitions/AtomicBTreePartition.java |  2 +-
 .../org/apache/cassandra/index/StubIndex.java   | 14 ++
 .../index/internal/CustomIndexTest.java | 29 
 3 files changed, 38 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java 
b/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
index ae8a1c3..2be882e 100644
--- a/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AtomicBTreePartition.java
@@ -127,7 +127,7 @@ public class AtomicBTreePartition extends 
AbstractBTreePartition
 updater.ref = current;
 updater.reset();
 
-if (!update.deletionInfo().isLive())
+if (!update.deletionInfo().getPartitionDeletion().isLive())
 
indexer.onPartitionDeletion(update.deletionInfo().getPartitionDeletion());
 
 if (update.deletionInfo().hasRanges())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/test/unit/org/apache/cassandra/index/StubIndex.java
--
diff --git a/test/unit/org/apache/cassandra/index/StubIndex.java 
b/test/unit/org/apache/cassandra/index/StubIndex.java
index 0d3cf47..0ea03dc 100644
--- a/test/unit/org/apache/cassandra/index/StubIndex.java
+++ b/test/unit/org/apache/cassandra/index/StubIndex.java
@@ -38,6 +38,8 @@ import org.apache.cassandra.utils.concurrent.OpOrder;
 
 public class StubIndex implements Index
 {
+public List partitionDeletions = new ArrayList<>();
+public List rangeTombstones = new ArrayList<>();
 public List rowsInserted = new ArrayList<>();
 public List rowsDeleted = new ArrayList<>();
 public List> rowsUpdated = new ArrayList<>();
@@ -48,6 +50,8 @@ public class StubIndex implements Index
 {
 rowsInserted.clear();
 rowsDeleted.clear();
+partitionDeletions.clear();
+rangeTombstones.clear();
 }
 
 public StubIndex(ColumnFamilyStore baseCfs, IndexMetadata metadata)
@@ -58,11 +62,7 @@ public class StubIndex implements Index
 
 public boolean indexes(PartitionColumns columns)
 {
-for (ColumnDefinition col : columns)
-for (ColumnIdentifier indexed : indexMetadata.columns)
-if (indexed.equals(col.name))
-return true;
-return false;
+return true;
 }
 
 public boolean shouldBuildBlocking()
@@ -80,7 +80,7 @@ public class StubIndex implements Index
 return filter;
 }
 
-public Indexer indexerFor(DecoratedKey key,
+public Indexer indexerFor(final DecoratedKey key,
   int nowInSec,
   OpOrder.Group opGroup,
   IndexTransaction.Type transactionType)
@@ -93,10 +93,12 @@ public class StubIndex implements Index
 
 public void partitionDelete(DeletionTime deletionTime)
 {
+partitionDeletions.add(deletionTime);
 }
 
 public void rangeTombstone(RangeTombstone tombstone)
 {
+rangeTombstones.add(tombstone);
 }
 
 public void insertRow(Row row)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c694cd62/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
--
diff --git a/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java 
b/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
index 255af4b..2f7a10b 100644
--- a/test/unit/org/apache/cassandra/index/internal/CustomIndexTest.java
+++ 

[jira] [Comment Edited] (CASSANDRA-10322) skipBytes is used extensively, but is slow

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747127#comment-14747127
 ] 

Benedict edited comment on CASSANDRA-10322 at 9/16/15 8:52 AM:
---

Certainly I'm not wed to the current approach, but we skipBytes _a lot_, and we 
have to have a loop somewhere - having the loop inside of the virtual 
invocation makes the most sense. What I've effectively done here is strengthen 
the contract of {{skipBytes}} in {{DataInputPlus}} - but I haven't declared 
this in the interface (which at the time I was a little torn on - given every 
single {{DataInput}} we've used has always honoured this stronger contract), so 
I would like to suggest either:

# Redeclaring {{skipBytes}} in {{DataInputPlus}} and making it clear that any 
implementor must skip fully unless EOF is reached _only_
# Declaring a new {{skipBytesFully}} in {{DataInputPlus}}

I prefer the former, as it is less code; I just (sloppily, in retrospect) opted 
not to actually redeclare it.


was (Author: benedict):
Certainly I'm not wed to the current approach, but we skipBytes _a lot_, and we 
have to have a loop somewhere - having the loop inside of the virtual 
invocation makes the most sense. What I've effectively done here is strengthen 
the contract of {{skipBytes}} in {{DataInputPlus}} - but I haven't declared 
this in the interface (which at the time I was a little torn on - given every 
single {{DataInput}} we've used has always honoured this stronger contract), so 
I would like to suggest either:

# Redeclaring {{skipBytes}} in {{DataInputPlus}} and making it clear that any 
implementor must skip fully unless EOF is reached _only_
# Declaring a new {{skipBytesFully}} in {{DataInputPlus}}

I prefer the former, as it is less code; I just opted not to actually redeclare 
it.

> skipBytes is used extensively, but is slow
> --
>
> Key: CASSANDRA-10322
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10322
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.x
>
>
> We skip a great deal to avoid materializing data. Ironically, however, 
> skipping is just as (perhaps more) expensive, as it allocates a temporary 
> array of the size of the number of bytes we want to skip.
> This trivial patch implements {{skipBytes}} more efficiently, and simplifies 
> {{FileUtils.skipBytesFully}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Benedict (JIRA)]

Benedict created CASSANDRA-10351:


 Summary: We should use unsignedVInt, instead of VInt methods, for 
size serializations
 Key: CASSANDRA-10351
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Trivial
 Fix For: 3.0.0 rc1


It's not a huge deal, but it means we more rapidly use an extra byte, and just 
as importantly we also have more work to do to interpret the data. Since we do 
this a lot, we may as well keep it as efficient as possible for scalar values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10330) Gossipinfo could return more useful information

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747236#comment-14747236
 ] 

Stefania commented on CASSANDRA-10330:
--

LGTM

> Gossipinfo could return more useful information
> ---
>
> Key: CASSANDRA-10330
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10330
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Minor
> Fix For: 2.1.x, 2.2.x, 2.0.17, 3.0.0 rc1
>
> Attachments: 10330.txt
>
>
> For instance, the version for each state, which can be useful for diagnosing 
> the reason for any missing states.  Also instead of just omitting the TOKENS 
> state, let's indicate whether the state was actually present or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9966) Option to apply statements within a batch sequentially

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747237#comment-14747237
 ] 

Benedict commented on CASSANDRA-9966:
-

We could always have a special resolution process, since here we know the 
"temporal" (sequential) ordering unambiguously.

It might be said this is ugly, since it is inconsistent with our normal 
resolution, but in many ways this is more consistent with the behaviour a user 
would receive applying the statements sequentially, even within the same 
millisecond, since the client state we keep server side increments the 
timestamp (if necessary) during execution. So simulating this behaviour without 
having to actually increment the timestamp seems to be the better of both 
worlds.

It doesn't leave me super happy to have another behavioural edge case. We 
already have more of these than we want, though, since drivers providing their 
own timestamps will each presumably have their own behaviour here. I think we 
should perhaps formalise the correct behaviour, make it part of any driver API, 
and make ourselves as consistent with the formalisation in each location as 
possible.

> Option to apply statements within a batch sequentially
> --
>
> Key: CASSANDRA-9966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9966
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Sam Overton
> Fix For: 3.x, 2.2.x
>
>
> It is possible to batch CAS statements such that their outcome is different 
> to the outcome were they executed sequentially outside of a batch.
> eg.
> a | b | c
> a | 1 | 1
> BEGIN BATCH
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> APPLY BATCH
> results in 
> a | b | c
> a | 2 | 2
> If these statements were not batched, the outcome would be 
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> a | b | c
> a | 2 | 1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> applied=false (pre-condition b=1 not met)
> Cassandra already checks for incompatible preconditions within a batch (eg 
> one statement with IF c=1 and another statement with IF c=2). It should also 
> check for mutations to columns in one statement that affect the 
> pre-conditions of another statement, or it should evaluate the statement 
> pre-conditions sequentially after applying the mutations of the previous 
> statement to an in-memory model of the partition.
> For backwards compatibility this would have to be a new "strict" batch mode, 
> eg.
> BEGIN STRICT BATCH



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8933) Short reads can return deleted results

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne resolved CASSANDRA-8933.
-
Resolution: Not A Problem

As said in my previous comment, it appears this is actually not a problem.

> Short reads can return deleted results
> --
>
> Key: CASSANDRA-8933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8933
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>
> The current code for short reads protection does not handle all cases.  
> Currently, we retry only if a node had returned the requested number of 
> results, but we have less results than that post-reconciliation, because this 
> means the node in question may have more results it hadn't sent due to the 
> limit.
> Consider however 3 nodes A, B, C (RF=3), and following sequence of operations 
> (all done at QUORUM):
> # we write 1 and 2 in a partition: all nodes get it.
> # we delete 1: only A and C get it.
> # we delete 2: only B and C get it.
> # we read the first row in the partition (so with a LIMIT 1) and A and B 
> answer first.
> At the last step, A will return the tombstone for 1 and the value 2, while B 
> will return just 1. So post reconciliation, we'll return 2 (since A returned 
> it and we have no tombstone for it), while we should return nothing. This is 
> a short read situation: B stopped at 1 because it was asked only 1 result, 
> but that result didn't made it in the result and we need further results from 
> it.  However, Because 1 results is requested and we have 1 result 
> post-reconciliation, the short read retry won't kick in.
> In practice, the short read check should be generalized: if any node X 
> returns the requested number of results but any of those results gets skipped 
> post-reconciliation, we might have a short read. Basically, enforcing the 
> limit replica-side is optimistic and assumes that all results of that replica 
> will be used, and as soon as that assumption fails we should get back more 
> results.
> Implementing that generalized condition can probably be done in 
> RowDataResolver.scheduleRepairs by using the repair to know if a node has had 
> some of results skipped by reconciliation but we want to know if a full CQL 
> row has been skipped or not so this will probably force us to add some 
> recounting.
> I'll note that I've fixed this problem on my branch for CASSANDRA-8099 (where 
> this is both simpler and somewhat more efficient since short reads don't 
> retry full queries there), so if decide this is too risky to fix in 2.1, we 
> can possibly just mark this as duplicate of CASSANDRA-8099.
> Lastly, it shouldn't be too hard to extends our current short read dtests to 
> test for that case, but I haven't taken the time to do so yet 
> ([~philipthompson] do you think you can have a look at adding such test at 
> some point?).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10238) Consolidating racks violates the RF contract

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-10238:
--
Fix Version/s: 2.2.2
   2.1.10
   3.0.0 rc1
   2.0.17

> Consolidating racks violates the RF contract
> 
>
> Key: CASSANDRA-10238
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10238
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.0.17, 3.0.0 rc1, 2.1.10, 2.2.2
>
>
> I have only tested this on 2.0 so far, but I suspect it will affect multiple 
> versions.
> Repro:
> * create a datacenter with rf>1
> * create more than one rack in this datacenter
> * consolidate these racks into 1
> * getendpoints will reveal the RF in practice is 1, even though other tools 
> will report the original RF that was set
> Restarting Cassandra will resolve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/6] cassandra git commit: Document post-2.1 caching table options syntax

[blerer]

Document post-2.1 caching table options syntax

patch by Paulo Motta; reviewed by Benjamin Lerer for CASSANDRA-9964


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3b46e5bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3b46e5bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3b46e5bd

Branch: refs/heads/trunk
Commit: 3b46e5bda61c0944ec28d56c289954d9f34c418e
Parents: 2cd59b8
Author: Paulo Motta 
Authored: Wed Sep 16 10:11:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:11:47 2015 +0200

--
 doc/cql3/CQL.textile | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3b46e5bd/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index a29ee6c..648c39f 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -327,13 +327,12 @@ Table creation supports the following other @@:
 |@dclocal_read_repair_chance@ | _simple_ | 0   | The probability with 
which to query extra nodes (e.g. more nodes than required by the consistency 
level) belonging to the same data center than the read coordinator for the 
purpose of read repairs.|
 |@gc_grace_seconds@   | _simple_ | 864000  | Time to wait before 
garbage collecting tombstones (deletion markers).|
 |@bloom_filter_fp_chance@ | _simple_ | 0.00075 | The target 
probability of false positive of the sstable bloom filters. Said bloom filters 
will be sized to provide the provided probability (thus lowering this value 
impact the size of bloom filters in-memory and on-disk)|
-|@compaction@ | _map_| _see below_ | The compaction 
options to use, see below.|
-|@compression@| _map_| _see below_ | Compression options, 
see below. |
-|@caching@| _simple_ | keys_only   | Whether to cache keys 
("key cache") and/or rows ("row cache") for this table. Valid values are: 
@all@, @keys_only@, @rows_only@ and @none@. |
 |@default_time_to_live@   | _simple_ | 0   | The default 
expiration time ("TTL") in seconds for a table.|
+|@compaction@ | _map_| _see below_ | Compaction options, 
see "below":#compactionOptions.|
+|@compression@| _map_| _see below_ | Compression options, 
see "below":#compressionOptions.|
+|@caching@| _map_| _see below_ | Caching options, see 
"below":#cachingOptions.|
 
-
-h4(#compactionOptions). @compaction@ options
+h4(#compactionOptions). Compaction options
 
 The @compaction@ property must at least define the @'class'@ sub-option, that 
defines the compaction strategy class to use. The default supported class are 
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@ and 
@'DateTieredCompactionStrategy'@. Custom strategy can be provided by specifying 
the full class name as a "string constant":#constants. The rest of the 
sub-options depends on the chosen class. The sub-options supported by the 
default classes are:
 
@@ -353,13 +352,22 @@ The @compaction@ property must at least define the 
@'class'@ sub-option, that de
 | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365 
 | SSTables only containing data that is older than this will never be 
compacted. |
 
 
-For the @compression@ property, the following default sub-options are 
available:
+h4(#compressionOptions). Compression options
+
+For the @compression@ property, the following sub-options are available:
 
 |_. option  |_. default|_. description |
 | @sstable_compression@ | LZ4Compressor| The compression algorithm to use. 
Default compressor are: LZ4Compressor, SnappyCompressor and DeflateCompressor. 
Use an empty string (@''@) to disable compression. Custom compressor can be 
provided by specifying the full class name as a "string constant":#constants.|
 | @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 | @crc_check_chance@| 1.0  | When compression is enabled, each 
compressed block includes a checksum of that block for the purpose of detecting 
disk bitrot and avoiding the propagation of corruption to other replica. This 
option defines the probability with which those checksums are checked during 
read. By default they are always checked. Set to 0 to disable checksum checking 
and to 0.5 for instance to check them every other read|
 
+h4(#cachingOptions). 

[5/6] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d4e0cd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d4e0cd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d4e0cd2

Branch: refs/heads/trunk
Commit: 0d4e0cd22653ff7c3c1ef4a4b2c23c78524ecc42
Parents: af0678d e7c2952
Author: blerer 
Authored: Wed Sep 16 10:33:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:34:00 2015 +0200

--
 build.xml|  1 +
 doc/cql3/CQL.textile | 20 ++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d4e0cd2/build.xml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d4e0cd2/doc/cql3/CQL.textile
--
diff --cc doc/cql3/CQL.textile
index 0e04528,ce712be..60e3d5e
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@@ -362,14 -361,22 +361,23 @@@ The @compaction@ property must at leas
  | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365
  | SSTables only containing data that is older than this will never be 
compacted. |
  
  
- For the @compression@ property, the following default sub-options are 
available:
+ h4(#compressionOptions). Compression options
+ 
+ For the @compression@ property, the following sub-options are available:
  
 -|_. option  |_. default|_. description |
 -| @sstable_compression@ | LZ4Compressor| The compression algorithm to 
use. Default compressor are: LZ4Compressor, SnappyCompressor and 
DeflateCompressor. Use an empty string (@''@) to disable compression. Custom 
compressor can be provided by specifying the full class name as a "string 
constant":#constants.|
 -| @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 -| @crc_check_chance@| 1.0  | When compression is enabled, 
each compressed block includes a checksum of that block for the purpose of 
detecting disk bitrot and avoiding the propagation of corruption to other 
replica. This option defines the probability with which those checksums are 
checked during read. By default they are always checked. Set to 0 to disable 
checksum checking and to 0.5 for instance to check them every other read|
 +|_. option |_. default|_. description |
 +| @class@  | LZ4Compressor| The compression algorithm to 
use. Default compressor are: LZ4Compressor, SnappyCompressor and 
DeflateCompressor. Use @'enabled' : false@ to disable compression. Custom 
compressor can be provided by specifying the full class name as a "string 
constant":#constants.|
 +| @enabled@| true | By default compression is 
enabled. To disable it, set @enabled@ to @false@
 +| @chunk_length_in_kb@ | 64KB | On disk SSTables are 
compressed by block (to allow random reads). This defines the size (in KB) of 
said block. Bigger values may improve the compression rate, but increases the 
minimum size of data to be read from disk for a read |
 +| @crc_check_chance@   | 1.0  | When compression is enabled, 
each compressed block includes a checksum of that block for the purpose of 
detecting disk bitrot and avoiding the propagation of corruption to other 
replica. This option defines the probability with which those checksums are 
checked during read. By default they are always checked. Set to 0 to disable 
checksum checking and to 0.5 for instance to check them every other read|
  
+ h4(#cachingOptions). Caching options
+ 
+ For the @caching@ property, the following sub-options are available:
+ 
+ |_. option  |_. default|_. description |
+ |@keys@ | ALL   | Whether to cache keys ("key cache") for 
this table. Valid values are: @ALL@ and @NONE@.|
+ |@rows_per_partition@   | NONE   | The amount of rows to cache per partition 
("row cache"). If an integer @n@ is specified, the first @n@ queried rows of a 
partition will be cached. Other possible options are @ALL@, to cache all rows 
of a queried partition, or @NONE@ to disable row caching.|
  
  h4. Other considerations:
  

[6/6] cassandra git commit: Merge branch cassandra-3.0 into trunk

[blerer]

Merge branch cassandra-3.0 into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/61ce1b9a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/61ce1b9a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/61ce1b9a

Branch: refs/heads/trunk
Commit: 61ce1b9a71c30b7c394253894a333e94653f13eb
Parents: 1ecb7a4 0d4e0cd
Author: blerer 
Authored: Wed Sep 16 10:35:32 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:37:06 2015 +0200

--
 build.xml   |  1 +
 doc/cql3/CQL.textile| 20 -
 .../cassandra/hadoop/cql3/CqlInputFormat.java   | 23 +++-
 3 files changed, 32 insertions(+), 12 deletions(-)
--

[3/6] cassandra git commit: Add Benjamin Lerer to the developer list in build.xml

[blerer]

Add Benjamin Lerer to the developer list in build.xml

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8b4eb76
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8b4eb76
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8b4eb76

Branch: refs/heads/trunk
Commit: b8b4eb76c42a61b4f33aa57041dac5af2ba2480c
Parents: 3b46e5b
Author: blerer 
Authored: Wed Sep 16 10:18:18 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:18:18 2015 +0200

--
 build.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8b4eb76/build.xml
--
diff --git a/build.xml b/build.xml
index 20c26a8..c36c819 100644
--- a/build.xml
+++ b/build.xml
@@ -415,6 +415,7 @@
 
 
 
+
 
 
 

[1/6] cassandra git commit: Fix build

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1ecb7a4f9 -> 61ce1b9a7


Fix build


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af0678dd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af0678dd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af0678dd

Branch: refs/heads/trunk
Commit: af0678dd48389a05b0fffb89d21b4e2f9d91f9d6
Parents: 0afe4be
Author: Sylvain Lebresne 
Authored: Wed Sep 16 09:44:16 2015 +0200
Committer: Sylvain Lebresne 
Committed: Wed Sep 16 09:44:16 2015 +0200

--
 .../cassandra/hadoop/cql3/CqlInputFormat.java   | 23 +++-
 1 file changed, 17 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/af0678dd/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java 
b/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
index 7699f29..3f74c33 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java
@@ -19,19 +19,30 @@ package org.apache.cassandra.hadoop.cql3;
 
 import java.io.IOException;
 import java.util.*;
-import java.util.concurrent.*
+import java.util.concurrent.*;
+
+import com.datastax.driver.core.Host;
+import com.datastax.driver.core.Metadata;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Row;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.TokenRange;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import com.datastax.driver.core.*;
 import org.apache.cassandra.db.SystemKeyspace;
 import org.apache.cassandra.dht.*;
+import org.apache.cassandra.thrift.KeyRange;
 import org.apache.cassandra.hadoop.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.mapred.*;
-import org.apache.hadoop.mapreduce.*;
 
-import com.datastax.driver.core.Row;
 
 /**
  * Hadoop InputFormat allowing map/reduce against Cassandra rows within one 
ColumnFamily.

[jira] [Commented] (CASSANDRA-10277) Empty BATCH throws exception in Cassandra

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747158#comment-14747158
 ] 

Benjamin Lerer commented on CASSANDRA-10277:


It is looking good to me.
Waiting for the CI results.

> Empty BATCH throws exception in Cassandra
> -
>
> Key: CASSANDRA-10277
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10277
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Vovodroid
>Priority: Minor
> Fix For: 2.2.2
>
> Attachments: 10277.txt, emptybatch.patch, emptytest.patch
>
>
> Connect to Cassandra with cqlsh and perform empty BATCH statement:
> {code}BEGIN BATCH  APPLY BATCH;{code}
> Following exception is thrown and logged
> {code}
> ERROR 14:02:07 Unexpected exception during request
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
>   at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
>   at 
> org.apache.cassandra.cql3.statements.BatchStatement$Parsed.prepare(BatchStatement.java:514)
>  ~[main/:na]
> ...
> {code}
> It's caused because of access to batch statement list, even if it's empty.
> Proposed patch (for 2.2.0) is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747178#comment-14747178
 ] 

Benedict edited comment on CASSANDRA-10286 at 9/16/15 9:09 AM:
---

[~philipthompson] this job failed completely, for what looks like environmental 
reasons (missing the tmp dirs it tried to create). Could you look into it, to 
see what the problem is?

We need to commit this before Friday; if we cannot get CI working, we'll have 
to commit it tomorrow and let the normal windows CI check it out.


was (Author: benedict):
[~ptnapoleon] this job failed completely, for what looks like environmental 
reasons (missing the tmp dirs it tried to create). Could you look into it, to 
see what the problem is?

We need to commit this before Friday; if we cannot get CI working, we'll have 
to commit it tomorrow and let the normal windows CI check it out.

> Windows utest 3.0: 
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError: 
>   at 
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete 
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException: 
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
>   at 
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
> ~[na:1.8.0_51]
>   at 
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
>  ~[na:1.8.0_51]
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>  ~[na:1.8.0_51]
>   at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
>  Source) [main/:na]
>   at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
>  Source) [main/:na]
>   at 
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) 
> [na:1.8.0_51]
>   at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540) 
> [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) 
> [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
>  [na:1.8.0_51]
>   at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 
> [na:1.8.0_51]
>   at 
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) 
> [na:1.8.0_51]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
>  [main/:na]
>   at 
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
>  [main/:na]
>   at 
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294) 
> [main/:na]
>   at 
> 

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747217#comment-14747217
 ] 

Benedict commented on CASSANDRA-10351:
--

This does make me wonder if we named the methods badly, as nearly every use 
case was {{write/readVInt}}, but should have been {{write/readUnsignedVInt}} - 
I don't think a single use of the signed variety was warranted.

Patch available [here|https://github.com/belliottsmith/cassandra/tree/10351]. 
[~slebresne], would you like to review (since I think you did the initial 
conversion)?

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10216) Remove target type from internal index metadata

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747080#comment-14747080
 ] 

Sylvain Lebresne commented on CASSANDRA-10216:
--

Provided cassci is happy, +1. Thanks.

> Remove target type from internal index metadata
> ---
>
> Key: CASSANDRA-10216
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10216
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>  Labels: client-impacting
> Fix For: 3.0.0 rc1
>
>
> As part of CASSANDRA-6716 & in anticipation of CASSANDRA-10124, a distinction 
> was introduced between secondary indexes which target a fixed set of 1 or 
> more columns in the base data, and those which are agnostic to the structure 
> of the underlying rows. This distinction is manifested in 
> {{IndexMetadata.targetType}} and {{system_schema.indexes}}, in the 
> {{target_type}} column. It could be argued that this distinction complicates 
> the codebase without providing any tangible benefit, given that the target 
> type is not actually used anywhere.
> It's only the impact on {{system_schema.indexes}} that makes puts this on the 
> critical path for 3.0, any code changes are just implementation details. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9966) Option to apply statements within a batch sequentially

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-9966:

Assignee: (was: Sylvain Lebresne)

> Option to apply statements within a batch sequentially
> --
>
> Key: CASSANDRA-9966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9966
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Sam Overton
> Fix For: 3.x, 2.2.x
>
>
> It is possible to batch CAS statements such that their outcome is different 
> to the outcome were they executed sequentially outside of a batch.
> eg.
> a | b | c
> a | 1 | 1
> BEGIN BATCH
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> APPLY BATCH
> results in 
> a | b | c
> a | 2 | 2
> If these statements were not batched, the outcome would be 
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> a | b | c
> a | 2 | 1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> applied=false (pre-condition b=1 not met)
> Cassandra already checks for incompatible preconditions within a batch (eg 
> one statement with IF c=1 and another statement with IF c=2). It should also 
> check for mutations to columns in one statement that affect the 
> pre-conditions of another statement, or it should evaluate the statement 
> pre-conditions sequentially after applying the mutations of the previous 
> statement to an in-memory model of the partition.
> For backwards compatibility this would have to be a new "strict" batch mode, 
> eg.
> BEGIN STRICT BATCH



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10020) Support eager retries for range queries

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10020:
-
Fix Version/s: 3.x

> Support eager retries for range queries
> ---
>
> Key: CASSANDRA-10020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10020
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Gautam Kumar
>Priority: Minor
> Fix For: 3.x
>
>
> A simple query like `select * from table` may time out if one of the nodes 
> fail.
> We had a 4-node cassandra cluster with RF=3 and CL=LOCAL_QUORUM. 
> The range query is issued to only two as per ConsistencyLevel.java: 
> liveEndpoints.subList(0, Math.min(liveEndpoints.size(), blockFor(keyspace)));
> If a node fails amongst this sublist, the query will time out. Why don't you 
> issue range queries to all replicas? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10343) Erroneous partition deletion events are delivered to indexers

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747112#comment-14747112
 ] 

Sylvain Lebresne commented on CASSANDRA-10343:
--

+1

> Erroneous partition deletion events are delivered to indexers
> -
>
> Key: CASSANDRA-10343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10343
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.0 rc1
>
>
> In {{AtomicBTreePartition#addAllWithSizeDelta}} we check 
> {{update.deletionInfo().isLive()}} to determine whether the 
> {{PartitionDeletion}} should be passed to the supplied {{IndexTransaction}}. 
> This is incorrect and it results in the {{PartitionDeletion}} being passed on 
> if *either* a partition delete or any range tombstones are present in the 
> update.
> We need to change:
> {code}
>   if (!update.deletionInfo().isLive())
> {code}
> to
> {code}
>   if (!update.deletionInfo().getPartitionDeletion().isLive())
> {code}
> Note that the built-in indexes don't currently do anything with the 
> {{deletionTime}} and so aren't affected, but any custom implementation which 
> didn't examine it properly might remove data from its index where it shouldn't



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10020) Support eager retries for range queries

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10020:
-
Reviewer:   (was: Sylvain Lebresne)

> Support eager retries for range queries
> ---
>
> Key: CASSANDRA-10020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10020
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Gautam Kumar
>Priority: Minor
> Fix For: 3.x
>
>
> A simple query like `select * from table` may time out if one of the nodes 
> fail.
> We had a 4-node cassandra cluster with RF=3 and CL=LOCAL_QUORUM. 
> The range query is issued to only two as per ConsistencyLevel.java: 
> liveEndpoints.subList(0, Math.min(liveEndpoints.size(), blockFor(keyspace)));
> If a node fails amongst this sublist, the query will time out. Why don't you 
> issue range queries to all replicas? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] cassandra git commit: Document post-2.1 caching table options syntax

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 8cad73290 -> e7c2952d3


Document post-2.1 caching table options syntax

patch by Paulo Motta; reviewed by Benjamin Lerer for CASSANDRA-9964


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3b46e5bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3b46e5bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3b46e5bd

Branch: refs/heads/cassandra-2.2
Commit: 3b46e5bda61c0944ec28d56c289954d9f34c418e
Parents: 2cd59b8
Author: Paulo Motta 
Authored: Wed Sep 16 10:11:47 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:11:47 2015 +0200

--
 doc/cql3/CQL.textile | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3b46e5bd/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index a29ee6c..648c39f 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -327,13 +327,12 @@ Table creation supports the following other @@:
 |@dclocal_read_repair_chance@ | _simple_ | 0   | The probability with 
which to query extra nodes (e.g. more nodes than required by the consistency 
level) belonging to the same data center than the read coordinator for the 
purpose of read repairs.|
 |@gc_grace_seconds@   | _simple_ | 864000  | Time to wait before 
garbage collecting tombstones (deletion markers).|
 |@bloom_filter_fp_chance@ | _simple_ | 0.00075 | The target 
probability of false positive of the sstable bloom filters. Said bloom filters 
will be sized to provide the provided probability (thus lowering this value 
impact the size of bloom filters in-memory and on-disk)|
-|@compaction@ | _map_| _see below_ | The compaction 
options to use, see below.|
-|@compression@| _map_| _see below_ | Compression options, 
see below. |
-|@caching@| _simple_ | keys_only   | Whether to cache keys 
("key cache") and/or rows ("row cache") for this table. Valid values are: 
@all@, @keys_only@, @rows_only@ and @none@. |
 |@default_time_to_live@   | _simple_ | 0   | The default 
expiration time ("TTL") in seconds for a table.|
+|@compaction@ | _map_| _see below_ | Compaction options, 
see "below":#compactionOptions.|
+|@compression@| _map_| _see below_ | Compression options, 
see "below":#compressionOptions.|
+|@caching@| _map_| _see below_ | Caching options, see 
"below":#cachingOptions.|
 
-
-h4(#compactionOptions). @compaction@ options
+h4(#compactionOptions). Compaction options
 
 The @compaction@ property must at least define the @'class'@ sub-option, that 
defines the compaction strategy class to use. The default supported class are 
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@ and 
@'DateTieredCompactionStrategy'@. Custom strategy can be provided by specifying 
the full class name as a "string constant":#constants. The rest of the 
sub-options depends on the chosen class. The sub-options supported by the 
default classes are:
 
@@ -353,13 +352,22 @@ The @compaction@ property must at least define the 
@'class'@ sub-option, that de
 | @max_sstable_age_days@   | DateTieredCompactionStrategy| 365 
 | SSTables only containing data that is older than this will never be 
compacted. |
 
 
-For the @compression@ property, the following default sub-options are 
available:
+h4(#compressionOptions). Compression options
+
+For the @compression@ property, the following sub-options are available:
 
 |_. option  |_. default|_. description |
 | @sstable_compression@ | LZ4Compressor| The compression algorithm to use. 
Default compressor are: LZ4Compressor, SnappyCompressor and DeflateCompressor. 
Use an empty string (@''@) to disable compression. Custom compressor can be 
provided by specifying the full class name as a "string constant":#constants.|
 | @chunk_length_kb@ | 64KB | On disk SSTables are compressed 
by block (to allow random reads). This defines the size (in KB) of said block. 
Bigger values may improve the compression rate, but increases the minimum size 
of data to be read from disk for a read |
 | @crc_check_chance@| 1.0  | When compression is enabled, each 
compressed block includes a checksum of that block for the purpose of detecting 
disk bitrot and avoiding the propagation of corruption to other replica. This 
option defines the probability with which those checksums are checked during 
read. By default they are always checked. Set to 0 to disable 

[3/3] cassandra git commit: Merge branch cassandra-2.1 into cassandra-2.2

[blerer]

Merge branch cassandra-2.1 into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e7c2952d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e7c2952d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e7c2952d

Branch: refs/heads/cassandra-2.2
Commit: e7c2952d31e8b6ddfef220942ba8306beabb
Parents: 8cad732 b8b4eb7
Author: blerer 
Authored: Wed Sep 16 10:21:44 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:21:58 2015 +0200

--
 build.xml|  1 +
 doc/cql3/CQL.textile | 20 ++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/build.xml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e7c2952d/doc/cql3/CQL.textile
--

[2/3] cassandra git commit: Add Benjamin Lerer to the developer list in build.xml

[blerer]

Add Benjamin Lerer to the developer list in build.xml

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8b4eb76
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8b4eb76
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8b4eb76

Branch: refs/heads/cassandra-2.2
Commit: b8b4eb76c42a61b4f33aa57041dac5af2ba2480c
Parents: 3b46e5b
Author: blerer 
Authored: Wed Sep 16 10:18:18 2015 +0200
Committer: blerer 
Committed: Wed Sep 16 10:18:18 2015 +0200

--
 build.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8b4eb76/build.xml
--
diff --git a/build.xml b/build.xml
index 20c26a8..c36c819 100644
--- a/build.xml
+++ b/build.xml
@@ -415,6 +415,7 @@
 
 
 
+
 
 
 

[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces

[Roman Bielik (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747206#comment-14747206
 ] 

Roman Bielik commented on CASSANDRA-9748:
-

Retested with the suggested settings, but it had no effect. The result is still 
the same.

How does "prefer_local" effect the gossip? I mean how does Cassandra identify 
which interface is local?

> Can't see other nodes when using multiple network interfaces
> 
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>
> The idea is to setup a multi-DC environment across 2 different networks based 
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1: 
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all 
> nodes can see each other (this one has a higher latency). 
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess). 
> The nodetool ring/status shows only the local node but not the other ones 
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that 
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0  0 0.0.0.0:71990.0.0.0:*   
> LISTEN  3587/java   
> tcp0  0 10.0.1.1:9160   0.0.0.0:*   
> LISTEN  3587/java   
> tcp0  0 10.0.1.1:9042   0.0.0.0:*   
> LISTEN  3587/java   
> tcp0  0 10.0.1.1:7000   0.0.0.0:*   
> LISTEN  3587/java   
> tcp0  0 127.0.0.1:7199  127.0.0.1:52874 
> ESTABLISHED 3587/java   
> tcp0  0 10.0.1.1:7199   10.0.1.1:39650  
> ESTABLISHED 3587/java 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10351:
-
Reviewer: Sylvain Lebresne

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747239#comment-14747239
 ] 

Sylvain Lebresne commented on CASSANDRA-10351:
--

I agree with this, though I'll not that the only reason I switched to the 
signed version initially is that I didn't wanted to argue against [this 
comment|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/util/DataOutputPlus.java#L51-L57]
 at the time, and so [~aweisberg] might have a different opinion. Though again, 
I'm personally in favor of this (but let's maybe remove or at least tone down 
said comment, it's silly to have such discouragement for {{writeUnsignedVInt}} 
if we use it almost exclusively).

On the patch, the only place I personally wouldn't change is in 
{{IndexHelper}}.  The rational is in [this CASSANDRA-10232 
comment|https://issues.apache.org/jira/browse/CASSANDRA-10232?focusedCommentId=14727390=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14727390].

Nit: In {{Batch.java}}, there is a duplicated import.


> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] cassandra git commit: Fix consolidating racks violating the RF contract

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 b8b4eb76c -> 4b1d59e13


Fix consolidating racks violating the RF contract

patch by Stefania Alborghetti; reviewed by Blake Eggleston for
CASSANDRA-10238


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/257cdaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/257cdaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/257cdaa0

Branch: refs/heads/cassandra-2.1
Commit: 257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b
Parents: 0bb32f0
Author: Stefania Alborghetti 
Authored: Fri Sep 11 16:31:40 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:40:31 2015 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d4cc15f..3c47427 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.17
+ * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
  * Disallow decommission when node is in drained state (CASSANDRA-8741)
  * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
  * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java 
b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
index 4f822c6..745eeb8 100644
--- a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
@@ -69,6 +69,7 @@ public class PropertyFileSnitch extends 
AbstractNetworkTopologySnitch
 protected void runMayThrow() throws ConfigurationException
 {
 reloadConfiguration();
+StorageService.instance.updateTopology();
 }
 };
 ResourceWatcher.watch(SNITCH_PROPERTIES_FILENAME, runnable, 60 * 
1000);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java 
b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index a673c94..b1b25e8 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -78,14 +78,14 @@ public class TokenMetadata
 // Finally, note that recording the tokens of joining nodes in 
bootstrapTokens also
 // means we can detect and reject the addition of multiple nodes at the 
same token
 // before one becomes part of the ring.
-private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap();
+private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap<>();
 // (don't need to record Token here since it's still part of 
tokenToEndpointMap until it's done leaving)
-private final Set leavingEndpoints = new 
HashSet();
+private final Set leavingEndpoints = new HashSet<>();
 // this is a cache of the calculation from {tokenToEndpointMap, 
bootstrapTokens, leavingEndpoints}
-private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap>();
+private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap<>();
 
 // nodes which are migrating to the new tokens in the ring
-private final Set> movingEndpoints = new 
HashSet>();
+private final Set> movingEndpoints = new 
HashSet<>();
 
 /* Use this lock for manipulating the token map */
 private final ReadWriteLock lock = new ReentrantReadWriteLock(true);
@@ -121,7 +121,7 @@ public class TokenMetadata
 
 private ArrayList sortTokens()
 {
-return new ArrayList(tokenToEndpointMap.keySet());
+return new ArrayList<>(tokenToEndpointMap.keySet());
 }
 
 

[jira] [Updated] (CASSANDRA-10352) Paging with DISTINCT and IN can throw ClassCastException

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10352:
-
Fix Version/s: 2.2.x

> Paging with DISTINCT and IN can throw ClassCastException
> 
>
> Key: CASSANDRA-10352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10352
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x, 2.2.x
>
>
> The ctor of {{SliceQueryPager}} has the following code:
> {noformat}
> if (state != null)
> {
> // The only case where this could be a non-CellName Composite is 
> if it's Composites.EMPTY, but that's not
> // valid for PagingState.cellName, so we can safely cast to 
> CellName.
> lastReturned = (CellName) 
> cfm.comparator.fromByteBuffer(state.cellName);
> restoreState(state.remaining, true);
> }
> {noformat}
> The assumption that we can only get a {{CellName}} is invalid however. When 
> using a {{MultiPartitionPager}} (so when the query has a {{IN}} on the 
> partition key), it's possible for a page to stop at the end of one of the 
> underlying pager, and for that pager to be exhausted. If that's the case, the 
> {{PagingState}} returned will have the partition key of the next key to 
> retrieve, but a {{null}} cellname (since we haven't started to query that 
> next key). When the next page is queried, that {{PagingState}} will trigger a 
> {{ClassCastException}}.
> The [following 
> dtest|https://github.com/pcmanus/cassandra-dtest/commit/b98a112fc22f92aa965d2068345c708ae43b8a85]
>  reproduces this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10352) Paging with DISTINCT and IN can throw ClassCastException

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747324#comment-14747324
 ] 

Sylvain Lebresne commented on CASSANDRA-10352:
--

I've pushed a fix for this 
[here|https://github.com/pcmanus/cassandra/commits/10352]. With that, the dtest 
linked above passes.

> Paging with DISTINCT and IN can throw ClassCastException
> 
>
> Key: CASSANDRA-10352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10352
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x, 2.2.x
>
>
> The ctor of {{SliceQueryPager}} has the following code:
> {noformat}
> if (state != null)
> {
> // The only case where this could be a non-CellName Composite is 
> if it's Composites.EMPTY, but that's not
> // valid for PagingState.cellName, so we can safely cast to 
> CellName.
> lastReturned = (CellName) 
> cfm.comparator.fromByteBuffer(state.cellName);
> restoreState(state.remaining, true);
> }
> {noformat}
> The assumption that we can only get a {{CellName}} is invalid however. When 
> using a {{MultiPartitionPager}} (so when the query has a {{IN}} on the 
> partition key), it's possible for a page to stop at the end of one of the 
> underlying pager, and for that pager to be exhausted. If that's the case, the 
> {{PagingState}} returned will have the partition key of the next key to 
> retrieve, but a {{null}} cellname (since we haven't started to query that 
> next key). When the next page is queried, that {{PagingState}} will trigger a 
> {{ClassCastException}}.
> The [following 
> dtest|https://github.com/pcmanus/cassandra-dtest/commit/b98a112fc22f92aa965d2068345c708ae43b8a85]
>  reproduces this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10326) Performance is worse in 3.0

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747370#comment-14747370
 ] 

Benedict commented on CASSANDRA-10326:
--

Another data point 
[here|http://cstar.datastax.com/graph?stats=a63084c2-5c4e-11e5-9b80-42010af0688f=op_rate=1_user=1_aggregates=true=0=907.06=0=144032.9].
 This is for non-reversed queries, and compares multiple branches.

Notable is that the mixed workload when performing non-reversed lookups is in 
the same ballpark (but still slower). However reads are still quite a bit 
slower, especially those for a random lookup of a single row.

> Performance is worse in 3.0
> ---
>
> Key: CASSANDRA-10326
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10326
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benedict
>Priority: Critical
> Fix For: 3.0.x
>
>
> Performance is generally turning out to be worse after 8099, despite a number 
> of unrelated performance enhancements being delivered. This isn't entirely 
> unexpected, given a great deal of time was spent optimising the old code, 
> however things appear worse than we had hoped.
> My expectation was that workloads making extensive use of CQL constructs 
> would be faster post-8099, however the latest tests performed with very large 
> CQL rows, including use of collections, still exhibit performance below that 
> of 2.1 and 2.2. 
> Eventually, as the dataset size grows large enough and the locality of access 
> is just right, the reduction in size of our dataset will yield a window 
> during which some users will perform better due simply to improved page cache 
> hit rates. We seem to see this in some of the tests. However we should be at 
> least as fast (and really faster) off the bat.
> The following are some large partition benchmark results, with as many as 40K 
> rows per partition, running LCS. There are a number of parameters we can 
> modify to see how behaviour changes and under what scenarios we might still 
> be faster, but the picture painted isn't brilliant, and is consistent, so we 
> should really try and figure out what's up before GA.
> [trades-with-flags (collections), 
> blade11b|http://cstar.datastax.com/graph?stats=f0a17292-5a13-11e5-847a-42010af0688f=op_rate=1_user=1_aggregates=true=0=4387.02=0=122951.4]
> [trades-with-flags (collections), 
> blade11|http://cstar.datastax.com/graph?stats=e250-5a13-11e5-ae0d-42010af0688f=op_rate=1_user=1_aggregates=true=0=4424.75=0=130158.6]
> [trades (no collections), 
> blade11|http://cstar.datastax.com/graph?stats=9b7da48e-570c-11e5-90fe-42010af0688f=op_rate=1_user=1_aggregates=true=0=2682.46=0=142547.9]
> [~slebresne]: will you have time to look into this before GA?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Migrate all size serialization to unsigned vint encoded

[benedict]

Migrate all size serialization to unsigned vint encoded

Unsigned vint encoding is more efficient in both CPU and disk burden,
so this patch employs it wherever sensible.

patch by benedict; reviewed by sylvain for CASSANDRA-10351


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/649a106c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/649a106c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/649a106c

Branch: refs/heads/trunk
Commit: 649a106c39b6a166c988fb647eaa33341e7371c4
Parents: 8134f09
Author: Benedict Elliott Smith 
Authored: Wed Sep 16 10:33:22 2015 +0100
Committer: Benedict Elliott Smith 
Committed: Wed Sep 16 13:16:47 2015 +0100

--
 .../org/apache/cassandra/batchlog/Batch.java| 16 -
 src/java/org/apache/cassandra/db/Columns.java   |  6 ++--
 src/java/org/apache/cassandra/db/Mutation.java  |  6 ++--
 .../org/apache/cassandra/db/ReadCommand.java|  6 ++--
 .../cassandra/db/SerializationHeader.java   | 36 ++--
 src/java/org/apache/cassandra/db/Slices.java|  6 ++--
 src/java/org/apache/cassandra/db/TypeSizes.java |  2 +-
 .../db/filter/ClusteringIndexNamesFilter.java   |  6 ++--
 .../cassandra/db/filter/ColumnFilter.java   |  6 ++--
 .../apache/cassandra/db/filter/DataLimits.java  | 30 
 .../apache/cassandra/db/filter/RowFilter.java   |  6 ++--
 .../apache/cassandra/db/rows/EncodingStats.java | 18 +-
 .../rows/UnfilteredRowIteratorSerializer.java   |  6 ++--
 .../cassandra/hints/EncodedHintMessage.java |  4 +--
 src/java/org/apache/cassandra/hints/Hint.java   |  8 ++---
 .../org/apache/cassandra/hints/HintMessage.java |  6 ++--
 .../cassandra/io/util/DataOutputPlus.java   |  8 ++---
 .../apache/cassandra/utils/ByteBufferUtil.java  |  8 ++---
 18 files changed, 92 insertions(+), 92 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/649a106c/src/java/org/apache/cassandra/batchlog/Batch.java
--
diff --git a/src/java/org/apache/cassandra/batchlog/Batch.java 
b/src/java/org/apache/cassandra/batchlog/Batch.java
index caa2682..e91e3ca 100644
--- a/src/java/org/apache/cassandra/batchlog/Batch.java
+++ b/src/java/org/apache/cassandra/batchlog/Batch.java
@@ -30,7 +30,7 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.UUIDSerializer;
 
 import static org.apache.cassandra.db.TypeSizes.sizeof;
-import static org.apache.cassandra.db.TypeSizes.sizeofVInt;
+import static org.apache.cassandra.db.TypeSizes.sizeofUnsignedVInt;
 
 public final class Batch
 {
@@ -87,11 +87,11 @@ public final class Batch
 long size = UUIDSerializer.serializer.serializedSize(batch.id, 
version);
 size += sizeof(batch.creationTime);
 
-size += sizeofVInt(batch.decodedMutations.size());
+size += sizeofUnsignedVInt(batch.decodedMutations.size());
 for (Mutation mutation : batch.decodedMutations)
 {
 int mutationSize = (int) 
Mutation.serializer.serializedSize(mutation, version);
-size += sizeofVInt(mutationSize);
+size += sizeofUnsignedVInt(mutationSize);
 size += mutationSize;
 }
 
@@ -105,10 +105,10 @@ public final class Batch
 UUIDSerializer.serializer.serialize(batch.id, out, version);
 out.writeLong(batch.creationTime);
 
-out.writeVInt(batch.decodedMutations.size());
+out.writeUnsignedVInt(batch.decodedMutations.size());
 for (Mutation mutation : batch.decodedMutations)
 {
-out.writeVInt(Mutation.serializer.serializedSize(mutation, 
version));
+
out.writeUnsignedVInt(Mutation.serializer.serializedSize(mutation, version));
 Mutation.serializer.serialize(mutation, out, version);
 }
 }
@@ -129,7 +129,7 @@ public final class Batch
 
 private static Collection 
readEncodedMutations(DataInputPlus in) throws IOException
 {
-int count = (int) in.readVInt();
+int count = (int) in.readUnsignedVInt();
 
 ArrayList mutations = new ArrayList<>(count);
 for (int i = 0; i < count; i++)
@@ -140,12 +140,12 @@ public final class Batch
 
 private static Collection decodeMutations(DataInputPlus in, 
int version) throws IOException
 {
-int count = (int) in.readVInt();
+int count = (int) in.readUnsignedVInt();
 
 ArrayList mutations = new ArrayList<>(count);
 for (int i = 0; i < count; i++)
 {
-in.readVInt(); // skip mutation size
+

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[benedict]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3ec5b45
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3ec5b45
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3ec5b45

Branch: refs/heads/trunk
Commit: f3ec5b456044713b8fcc409f5144969d9781e923
Parents: f3ff8df 649a106
Author: Benedict Elliott Smith 
Authored: Wed Sep 16 13:17:14 2015 +0100
Committer: Benedict Elliott Smith 
Committed: Wed Sep 16 13:17:14 2015 +0100

--
 .../org/apache/cassandra/batchlog/Batch.java| 16 -
 src/java/org/apache/cassandra/db/Columns.java   |  6 ++--
 src/java/org/apache/cassandra/db/Mutation.java  |  6 ++--
 .../org/apache/cassandra/db/ReadCommand.java|  6 ++--
 .../cassandra/db/SerializationHeader.java   | 36 ++--
 src/java/org/apache/cassandra/db/Slices.java|  6 ++--
 src/java/org/apache/cassandra/db/TypeSizes.java |  2 +-
 .../db/filter/ClusteringIndexNamesFilter.java   |  6 ++--
 .../cassandra/db/filter/ColumnFilter.java   |  6 ++--
 .../apache/cassandra/db/filter/DataLimits.java  | 30 
 .../apache/cassandra/db/filter/RowFilter.java   |  6 ++--
 .../apache/cassandra/db/rows/EncodingStats.java | 18 +-
 .../rows/UnfilteredRowIteratorSerializer.java   |  6 ++--
 .../cassandra/hints/EncodedHintMessage.java |  4 +--
 src/java/org/apache/cassandra/hints/Hint.java   |  8 ++---
 .../org/apache/cassandra/hints/HintMessage.java |  6 ++--
 .../cassandra/io/util/DataOutputPlus.java   |  8 ++---
 .../apache/cassandra/utils/ByteBufferUtil.java  |  8 ++---
 18 files changed, 92 insertions(+), 92 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3ec5b45/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
--

[1/3] cassandra git commit: Migrate all size serialization to unsigned vint encoded

[benedict]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 8134f09c2 -> 649a106c3
  refs/heads/trunk f3ff8df05 -> f3ec5b456


Migrate all size serialization to unsigned vint encoded

Unsigned vint encoding is more efficient in both CPU and disk burden,
so this patch employs it wherever sensible.

patch by benedict; reviewed by sylvain for CASSANDRA-10351


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/649a106c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/649a106c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/649a106c

Branch: refs/heads/cassandra-3.0
Commit: 649a106c39b6a166c988fb647eaa33341e7371c4
Parents: 8134f09
Author: Benedict Elliott Smith 
Authored: Wed Sep 16 10:33:22 2015 +0100
Committer: Benedict Elliott Smith 
Committed: Wed Sep 16 13:16:47 2015 +0100

--
 .../org/apache/cassandra/batchlog/Batch.java| 16 -
 src/java/org/apache/cassandra/db/Columns.java   |  6 ++--
 src/java/org/apache/cassandra/db/Mutation.java  |  6 ++--
 .../org/apache/cassandra/db/ReadCommand.java|  6 ++--
 .../cassandra/db/SerializationHeader.java   | 36 ++--
 src/java/org/apache/cassandra/db/Slices.java|  6 ++--
 src/java/org/apache/cassandra/db/TypeSizes.java |  2 +-
 .../db/filter/ClusteringIndexNamesFilter.java   |  6 ++--
 .../cassandra/db/filter/ColumnFilter.java   |  6 ++--
 .../apache/cassandra/db/filter/DataLimits.java  | 30 
 .../apache/cassandra/db/filter/RowFilter.java   |  6 ++--
 .../apache/cassandra/db/rows/EncodingStats.java | 18 +-
 .../rows/UnfilteredRowIteratorSerializer.java   |  6 ++--
 .../cassandra/hints/EncodedHintMessage.java |  4 +--
 src/java/org/apache/cassandra/hints/Hint.java   |  8 ++---
 .../org/apache/cassandra/hints/HintMessage.java |  6 ++--
 .../cassandra/io/util/DataOutputPlus.java   |  8 ++---
 .../apache/cassandra/utils/ByteBufferUtil.java  |  8 ++---
 18 files changed, 92 insertions(+), 92 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/649a106c/src/java/org/apache/cassandra/batchlog/Batch.java
--
diff --git a/src/java/org/apache/cassandra/batchlog/Batch.java 
b/src/java/org/apache/cassandra/batchlog/Batch.java
index caa2682..e91e3ca 100644
--- a/src/java/org/apache/cassandra/batchlog/Batch.java
+++ b/src/java/org/apache/cassandra/batchlog/Batch.java
@@ -30,7 +30,7 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.UUIDSerializer;
 
 import static org.apache.cassandra.db.TypeSizes.sizeof;
-import static org.apache.cassandra.db.TypeSizes.sizeofVInt;
+import static org.apache.cassandra.db.TypeSizes.sizeofUnsignedVInt;
 
 public final class Batch
 {
@@ -87,11 +87,11 @@ public final class Batch
 long size = UUIDSerializer.serializer.serializedSize(batch.id, 
version);
 size += sizeof(batch.creationTime);
 
-size += sizeofVInt(batch.decodedMutations.size());
+size += sizeofUnsignedVInt(batch.decodedMutations.size());
 for (Mutation mutation : batch.decodedMutations)
 {
 int mutationSize = (int) 
Mutation.serializer.serializedSize(mutation, version);
-size += sizeofVInt(mutationSize);
+size += sizeofUnsignedVInt(mutationSize);
 size += mutationSize;
 }
 
@@ -105,10 +105,10 @@ public final class Batch
 UUIDSerializer.serializer.serialize(batch.id, out, version);
 out.writeLong(batch.creationTime);
 
-out.writeVInt(batch.decodedMutations.size());
+out.writeUnsignedVInt(batch.decodedMutations.size());
 for (Mutation mutation : batch.decodedMutations)
 {
-out.writeVInt(Mutation.serializer.serializedSize(mutation, 
version));
+
out.writeUnsignedVInt(Mutation.serializer.serializedSize(mutation, version));
 Mutation.serializer.serialize(mutation, out, version);
 }
 }
@@ -129,7 +129,7 @@ public final class Batch
 
 private static Collection 
readEncodedMutations(DataInputPlus in) throws IOException
 {
-int count = (int) in.readVInt();
+int count = (int) in.readUnsignedVInt();
 
 ArrayList mutations = new ArrayList<>(count);
 for (int i = 0; i < count; i++)
@@ -140,12 +140,12 @@ public final class Batch
 
 private static Collection decodeMutations(DataInputPlus in, 
int version) throws IOException
 {
-int count = (int) in.readVInt();
+int count = (int) in.readUnsignedVInt();
 
 ArrayList mutations = new 

[jira] [Commented] (CASSANDRA-9966) Option to apply statements within a batch sequentially

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747267#comment-14747267
 ] 

Benedict commented on CASSANDRA-9966:
-

I mean when applying a batch we use special resolution rules of always taking 
the new value.

If we want to avoid surprising users we probably would need to implement a 
special resolution of only incrementing the timestamp on a conflicting 
resolution, as otherwise users with more than 1000 batch statements would see a 
behavioural change that may (possibly) break things? And when applying 
individual statements we do effectively resolve this way (just with timestamp 
increments). So in order to minimise surprise, it seems simulating timestamp 
increments, without actually incrementing them (since we have the necessary 
information to do so) seems perhaps the least surprising option.

> Option to apply statements within a batch sequentially
> --
>
> Key: CASSANDRA-9966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9966
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Sam Overton
> Fix For: 3.x, 2.2.x
>
>
> It is possible to batch CAS statements such that their outcome is different 
> to the outcome were they executed sequentially outside of a batch.
> eg.
> a | b | c
> a | 1 | 1
> BEGIN BATCH
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> APPLY BATCH
> results in 
> a | b | c
> a | 2 | 2
> If these statements were not batched, the outcome would be 
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> a | b | c
> a | 2 | 1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> applied=false (pre-condition b=1 not met)
> Cassandra already checks for incompatible preconditions within a batch (eg 
> one statement with IF c=1 and another statement with IF c=2). It should also 
> check for mutations to columns in one statement that affect the 
> pre-conditions of another statement, or it should evaluate the statement 
> pre-conditions sequentially after applying the mutations of the previous 
> statement to an in-memory model of the partition.
> For backwards compatibility this would have to be a new "strict" batch mode, 
> eg.
> BEGIN STRICT BATCH



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Fix consolidating racks violating the RF contract

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.0 0bb32f0b6 -> 257cdaa08


Fix consolidating racks violating the RF contract

patch by Stefania Alborghetti; reviewed by Blake Eggleston for
CASSANDRA-10238


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/257cdaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/257cdaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/257cdaa0

Branch: refs/heads/cassandra-2.0
Commit: 257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b
Parents: 0bb32f0
Author: Stefania Alborghetti 
Authored: Fri Sep 11 16:31:40 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:40:31 2015 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d4cc15f..3c47427 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.17
+ * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
  * Disallow decommission when node is in drained state (CASSANDRA-8741)
  * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
  * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java 
b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
index 4f822c6..745eeb8 100644
--- a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
@@ -69,6 +69,7 @@ public class PropertyFileSnitch extends 
AbstractNetworkTopologySnitch
 protected void runMayThrow() throws ConfigurationException
 {
 reloadConfiguration();
+StorageService.instance.updateTopology();
 }
 };
 ResourceWatcher.watch(SNITCH_PROPERTIES_FILENAME, runnable, 60 * 
1000);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java 
b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index a673c94..b1b25e8 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -78,14 +78,14 @@ public class TokenMetadata
 // Finally, note that recording the tokens of joining nodes in 
bootstrapTokens also
 // means we can detect and reject the addition of multiple nodes at the 
same token
 // before one becomes part of the ring.
-private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap();
+private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap<>();
 // (don't need to record Token here since it's still part of 
tokenToEndpointMap until it's done leaving)
-private final Set leavingEndpoints = new 
HashSet();
+private final Set leavingEndpoints = new HashSet<>();
 // this is a cache of the calculation from {tokenToEndpointMap, 
bootstrapTokens, leavingEndpoints}
-private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap>();
+private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap<>();
 
 // nodes which are migrating to the new tokens in the ring
-private final Set> movingEndpoints = new 
HashSet>();
+private final Set> movingEndpoints = new 
HashSet<>();
 
 /* Use this lock for manipulating the token map */
 private final ReadWriteLock lock = new ReentrantReadWriteLock(true);
@@ -121,7 +121,7 @@ public class TokenMetadata
 
 private ArrayList sortTokens()
 {
-return new ArrayList(tokenToEndpointMap.keySet());
+return new ArrayList<>(tokenToEndpointMap.keySet());
 }
 
 

[1/3] cassandra git commit: Fix consolidating racks violating the RF contract

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 e7c2952d3 -> 5b05b6826


Fix consolidating racks violating the RF contract

patch by Stefania Alborghetti; reviewed by Blake Eggleston for
CASSANDRA-10238


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/257cdaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/257cdaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/257cdaa0

Branch: refs/heads/cassandra-2.2
Commit: 257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b
Parents: 0bb32f0
Author: Stefania Alborghetti 
Authored: Fri Sep 11 16:31:40 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:40:31 2015 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d4cc15f..3c47427 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.17
+ * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
  * Disallow decommission when node is in drained state (CASSANDRA-8741)
  * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
  * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java 
b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
index 4f822c6..745eeb8 100644
--- a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
@@ -69,6 +69,7 @@ public class PropertyFileSnitch extends 
AbstractNetworkTopologySnitch
 protected void runMayThrow() throws ConfigurationException
 {
 reloadConfiguration();
+StorageService.instance.updateTopology();
 }
 };
 ResourceWatcher.watch(SNITCH_PROPERTIES_FILENAME, runnable, 60 * 
1000);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java 
b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index a673c94..b1b25e8 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -78,14 +78,14 @@ public class TokenMetadata
 // Finally, note that recording the tokens of joining nodes in 
bootstrapTokens also
 // means we can detect and reject the addition of multiple nodes at the 
same token
 // before one becomes part of the ring.
-private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap();
+private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap<>();
 // (don't need to record Token here since it's still part of 
tokenToEndpointMap until it's done leaving)
-private final Set leavingEndpoints = new 
HashSet();
+private final Set leavingEndpoints = new HashSet<>();
 // this is a cache of the calculation from {tokenToEndpointMap, 
bootstrapTokens, leavingEndpoints}
-private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap>();
+private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap<>();
 
 // nodes which are migrating to the new tokens in the ring
-private final Set> movingEndpoints = new 
HashSet>();
+private final Set> movingEndpoints = new 
HashSet<>();
 
 /* Use this lock for manipulating the token map */
 private final ReadWriteLock lock = new ReentrantReadWriteLock(true);
@@ -121,7 +121,7 @@ public class TokenMetadata
 
 private ArrayList sortTokens()
 {
-return new ArrayList(tokenToEndpointMap.keySet());
+return new ArrayList<>(tokenToEndpointMap.keySet());
 }
 
 

[jira] [Commented] (CASSANDRA-9966) Option to apply statements within a batch sequentially

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747289#comment-14747289
 ] 

Sylvain Lebresne commented on CASSANDRA-9966:
-

Oh, I got confused because "normal resolution" made me think of the resolution 
going on when reconciling memtable and sstable contents, and I didn't saw how 
we could ever deal with that at that time (without major internal changes at 
least). But I assume you mean when creating the initial {{Mutation}} from the 
{{BatchStatement}}, in which case, yes, that make sense.

> Option to apply statements within a batch sequentially
> --
>
> Key: CASSANDRA-9966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9966
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Sam Overton
> Fix For: 3.x, 2.2.x
>
>
> It is possible to batch CAS statements such that their outcome is different 
> to the outcome were they executed sequentially outside of a batch.
> eg.
> a | b | c
> a | 1 | 1
> BEGIN BATCH
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> APPLY BATCH
> results in 
> a | b | c
> a | 2 | 2
> If these statements were not batched, the outcome would be 
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> a | b | c
> a | 2 | 1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> applied=false (pre-condition b=1 not met)
> Cassandra already checks for incompatible preconditions within a batch (eg 
> one statement with IF c=1 and another statement with IF c=2). It should also 
> check for mutations to columns in one statement that affect the 
> pre-conditions of another statement, or it should evaluate the statement 
> pre-conditions sequentially after applying the mutations of the previous 
> statement to an in-memory model of the partition.
> For backwards compatibility this would have to be a new "strict" batch mode, 
> eg.
> BEGIN STRICT BATCH



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10352) Paging with DISTINCT and IN can throw ClassCastException

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747318#comment-14747318
 ] 

Sylvain Lebresne commented on CASSANDRA-10352:
--

Forget to mention that as far as I can tell, this can only happen when we only 
query static columns (hence the {{DISTINCT}} in the description and test) 
because I believe it's the only case where {{MultiPartitionPager}} can end up a 
pager with the last used sub-pager exhausted (and that's because {{DISTINCT}} 
with statics force each partition count to 1, and this even though the query 
limit is much larger (or has no limit)). And if I'm wrong and there is more 
ways to trigger that, then all the more reason to fix.

> Paging with DISTINCT and IN can throw ClassCastException
> 
>
> Key: CASSANDRA-10352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10352
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x
>
>
> The ctor of {{SliceQueryPager}} has the following code:
> {noformat}
> if (state != null)
> {
> // The only case where this could be a non-CellName Composite is 
> if it's Composites.EMPTY, but that's not
> // valid for PagingState.cellName, so we can safely cast to 
> CellName.
> lastReturned = (CellName) 
> cfm.comparator.fromByteBuffer(state.cellName);
> restoreState(state.remaining, true);
> }
> {noformat}
> The assumption that we can only get a {{CellName}} is invalid however. When 
> using a {{MultiPartitionPager}} (so when the query has a {{IN}} on the 
> partition key), it's possible for a page to stop at the end of one of the 
> underlying pager, and for that pager to be exhausted. If that's the case, the 
> {{PagingState}} returned will have the partition key of the next key to 
> retrieve, but a {{null}} cellname (since we haven't started to query that 
> next key). When the next page is queried, that {{PagingState}} will trigger a 
> {{ClassCastException}}.
> The [following 
> dtest|https://github.com/pcmanus/cassandra-dtest/commit/b98a112fc22f92aa965d2068345c708ae43b8a85]
>  reproduces this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7486) Migrate to G1GC by default

[Sebastian Martinka (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747334#comment-14747334
 ] 

Sebastian Martinka commented on CASSANDRA-7486:
---

For future tests you could use logstash and kibana to generate graphs from G1 
and CMS logs: https://github.com/Mortinke/logstash-pattern

> Migrate to G1GC by default
> --
>
> Key: CASSANDRA-7486
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7486
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Config
>Reporter: Jonathan Ellis
>Assignee: Albert P Tobey
> Fix For: 3.0 alpha 1
>
>
> See 
> http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning
>  and https://twitter.com/rbranson/status/482113561431265281
> May want to default 2.1 to G1.
> 2.1 is a different animal from 2.0 after moving most of memtables off heap.  
> Suspect this will help G1 even more than CMS.  (NB this is off by default but 
> needs to be part of the test.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10322) skipBytes is used extensively, but is slow

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747365#comment-14747365
 ] 

Benedict commented on CASSANDRA-10322:
--

For information, [this 
graph|http://cstar.datastax.com/graph?stats=a63084c2-5c4e-11e5-9b80-42010af0688f=op_rate=3_user=1_aggregates=true=0=174.24=0=110044]
 demonstrates a useful bump from this patch (the purple line). Of course, still 
not up to 2.2 speed, and 2.2 could also have the optimisation if we wanted to 
backport it (which perhaps we should, so we have a good yardstick to continue 
to measure from).

> skipBytes is used extensively, but is slow
> --
>
> Key: CASSANDRA-10322
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10322
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.x
>
>
> We skip a great deal to avoid materializing data. Ironically, however, 
> skipping is just as (perhaps more) expensive, as it allocates a temporary 
> array of the size of the number of bytes we want to skip.
> This trivial patch implements {{skipBytes}} more efficiently, and simplifies 
> {{FileUtils.skipBytesFully}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14768836#comment-14768836
 ] 

Benedict commented on CASSANDRA-10351:
--

Committed as 649a106c39b6a166c988fb647eaa33341e7371c4.

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747258#comment-14747258
 ] 

Benedict commented on CASSANDRA-10351:
--

I've pushed a version with that addressed. I've got a follow up commit that 
also reintroduces a message to {{writeUnsignedVInt}}, if you think it adds 
informational value.

I've reverted {{IndexHelper}} - it's worth noting that we would have still 
serialized negative values just fine, but in case anyone did do this they would 
have incurred higher costs than we save by sticking with signed. Plus, it's 
nice for signed encoding to have a purpose in life.

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747276#comment-14747276
 ] 

Benedict commented on CASSANDRA-10351:
--

Right. I consider commit to be contingent on a (contributor +1, cassci +1) 
pair. Which is a bit of a problem sometimes; I've been waiting two days for a 
successful cassci run for CASSANDRA-10301, and longer for CASSANDRA-10286 (all 
environment issues, AFAICT).

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

[aleksey]

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4b1d59e1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4b1d59e1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4b1d59e1

Branch: refs/heads/cassandra-2.2
Commit: 4b1d59e13143f5a55478a7cfcaca61ba22259f9f
Parents: b8b4eb7 257cdaa
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:44:28 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:44:28 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 7 files changed, 333 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/CHANGES.txt
--
diff --cc CHANGES.txt
index 5f11049,3c47427..12523be
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,46 -1,7 +1,47 @@@
 -2.0.17
 +2.1.10
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * Added configurable warning threshold for GC duration (CASSANDRA-8907)
 + * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 + * Fix handling of streaming EOF (CASSANDRA-10206)
 + * Only check KeyCache when it is enabled
 + * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
 + * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 +Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
 - * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
 +
 +
 +2.1.9
 + * Avoid race condition during read repair (CASSANDRA-9460)
 + * (cqlsh) default load-from-file encoding to utf-8 (CASSANDRA-9898)
 + * Avoid returning Permission.NONE when failing to query users table 
(CASSANDRA-10168)
 + * (cqlsh) Allow encoding to be set through command line (CASSANDRA-10004)
 + * Add new JMX methods to change local compaction strategy (CASSANDRA-9965)
 + * Write hints for paxos commits (CASSANDRA-7342)
 + * (cqlsh) Fix timestamps before 1970 on Windows, always
 +   use UTC for timestamp display (CASSANDRA-1)
 + * (cqlsh) Avoid overwriting new config file with old config
 +   when both exist (CASSANDRA-9777)
 + * Release snapshot selfRef when doing snapshot repair (CASSANDRA-9998)
 + * Cannot replace token does not exist - DN node removed as Fat Client 
(CASSANDRA-9871)
 + * Fix handling of enable/disable autocompaction (CASSANDRA-9899)
 + * Commit log segment recycling is disabled by default (CASSANDRA-9896)
 + * Add consistency level to tracing ouput (CASSANDRA-9827)
 + * Fix MarshalException when upgrading superColumn family (CASSANDRA-9582)
 + * Fix broken logging for "empty" flushes in Memtable (CASSANDRA-9837)
 + * Handle corrupt files on startup (CASSANDRA-9686)
 + * Fix clientutil jar and tests (CASSANDRA-9760)
 + * (cqlsh) Allow the SSL protocol version to be specified through the
 +   config file or environment variables (CASSANDRA-9544)
 + * Remove repair snapshot leftover on startup (CASSANDRA-7357)
 + * Use random nodes for batch log when only 2 racks (CASSANDRA-8735)
 + * Ensure atomicity inside thrift and stream session (CASSANDRA-7757)
 + * Fix nodetool info error when the node is not joined (CASSANDRA-9031)
 +Merged from 2.0:
   * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)
   * Add tool to find why expired sstables are not getting dropped 
(CASSANDRA-10015)
   * Remove erroneous pending HH tasks from tpstats/jmx (CASSANDRA-9129)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
index 2c0980a,dd1637d..da43600
--- a/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
@@@ -72,24 -62,7 +72,25 @@@ public class GossipingPropertyFileSnitc
  }
  catch (ConfigurationException e)
 

[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5b05b682
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5b05b682
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5b05b682

Branch: refs/heads/cassandra-2.2
Commit: 5b05b682641f634d8b6548d225b9130b771df969
Parents: e7c2952 4b1d59e
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:48:56 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:48:56 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/CHANGES.txt
--
diff --cc CHANGES.txt
index b0ade42,12523be..7b5fa78
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -20,8 -10,8 +20,9 @@@ Merged from 2.1
   * Only check KeyCache when it is enabled
   * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
   * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 + * Add nodetool gettraceprobability command (CASSANDRA-10234)
  Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
  
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/TokenMetadata.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/service/StorageService.java
--

[5/5] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[aleksey]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3ff8df0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3ff8df0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3ff8df0

Branch: refs/heads/trunk
Commit: f3ff8df0543e24284806908e3861189b4c28e92a
Parents: 48da81d 8134f09
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 12:05:22 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 12:05:22 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 ++---
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 213 ++-
 6 files changed, 335 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3ff8df0/CHANGES.txt
--

[4/4] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aleksey]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8134f09c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8134f09c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8134f09c

Branch: refs/heads/cassandra-3.0
Commit: 8134f09c2bdf2d859135a187a9de91f4cfb7620d
Parents: c694cd6 5b05b68
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 12:04:53 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 12:04:53 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 ++---
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 213 ++-
 6 files changed, 335 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --cc src/java/org/apache/cassandra/locator/TokenMetadata.java
index c3ec353,8d54ead..75448a8
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@@ -124,18 -119,9 +124,18 @@@ public class TokenMetadat
  sortedTokens = sortTokens();
  }
  
 +/**
 + * To be used by tests only (via {@link 
StorageService.setPartitionerUnsafe}).
 + */
 +@VisibleForTesting
 +public TokenMetadata cloneWithNewPartitioner(IPartitioner newPartitioner)
 +{
 +return new TokenMetadata(tokenToEndpointMap, endpointToHostIdMap, 
topology, newPartitioner);
 +}
 +
  private ArrayList sortTokens()
  {
- return new ArrayList(tokenToEndpointMap.keySet());
+ return new ArrayList<>(tokenToEndpointMap.keySet());
  }
  
  /** @return the number of nodes bootstrapping into source's primary range 
*/
@@@ -533,10 -551,9 +565,10 @@@
  lock.readLock().lock();
  try
  {
- return new TokenMetadata(SortedBiMultiValMap.create(tokenToEndpointMap, null, inetaddressCmp),
+ return new 
TokenMetadata(SortedBiMultiValMap.create(tokenToEndpointMap, null, 
inetaddressCmp),
   HashBiMap.create(endpointToHostIdMap),
 - new Topology(topology));
 + new Topology(topology),
 + partitioner);
  }
  finally
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/src/java/org/apache/cassandra/service/StorageService.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
--
diff --cc test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
index 68d0dfa,fc8095d..e7bb70a
--- a/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
+++ b/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
@@@ -29,9 -42,6 +41,7 @@@ import org.apache.cassandra.config.Data
  import org.apache.cassandra.dht.Token;
  import org.apache.cassandra.service.StorageService;
  
- import static org.apache.cassandra.Util.token;
- import static org.junit.Assert.assertEquals;
 +
  @RunWith(OrderedJUnit4ClassRunner.class)
  public class TokenMetadataTest
  {

[jira] [Commented] (CASSANDRA-10238) Consolidating racks violates the RF contract

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747306#comment-14747306
 ] 

Aleksey Yeschenko commented on CASSANDRA-10238:
---

Committed as 
[257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b|https://github.com/apache/cassandra/commit/257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b]
 to cassandra-2.0 and merged upwards. Thank you.

> Consolidating racks violates the RF contract
> 
>
> Key: CASSANDRA-10238
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10238
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.0.17, 3.0.0 rc1, 2.1.10, 2.2.2
>
>
> I have only tested this on 2.0 so far, but I suspect it will affect multiple 
> versions.
> Repro:
> * create a datacenter with rf>1
> * create more than one rack in this datacenter
> * consolidate these racks into 1
> * getendpoints will reveal the RF in practice is 1, even though other tools 
> will report the original RF that was set
> Restarting Cassandra will resolve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9930) Add test coverage for 2.1 -> 3.0 upgrades

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-9930:
-
Fix Version/s: (was: 3.0.0 rc1)
   3.0.x

> Add test coverage for 2.1 -> 3.0 upgrades
> -
>
> Key: CASSANDRA-9930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9930
> Project: Cassandra
>  Issue Type: Test
>Reporter: Ryan McGuire
>Assignee: Russ Hatch
> Fix For: 3.0.x
>
>
> Once we move to a [tick-tock 
> model|http://www.mail-archive.com/dev%40cassandra.apache.org/msg07771.html] 
> we will need the ability to upgrade from stable release to stable release, 
> skipping a single version in the process.
> Example: 3.0 -> 3.2 (skipping the pass-through upgrade to 3.1 we normally 
> have to do)
> I would recommend we code this so that we can upgrade 2.1 directly to 3.0 as 
> well, it's probably too much work to do this for versions prior to that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10254) 3.0 paging states are incompatible with pre-3.0 nodes

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747336#comment-14747336
 ] 

Sylvain Lebresne commented on CASSANDRA-10254:
--

I've push a patch [here|https://github.com/pcmanus/cassandra/commits/10352] for 
fixing the incompatibility of paging state with pre-3.0 nodes. As said above, 
what the patch does is that is preserves the pre-existing format for paging 
states when using the native protocol v3. For the native protocol v4 however, 
we avoid it since 1) that saves us some work and 2) it allows to use a slightly 
more efficient format for the paging state.

That patch actually does _not_ fix 
{{TestCQL.static_columns_with_distinct_test}} and it took me a while to figure 
out why (as I assumed it was a 3.0 bug), but it turns out that it's an actual 
bug in 2.1/2.2 and I've opened CASSANDRA-10352 for that (and so mea culpa, I 
was wrong, this test failure is a 2.1/2.2 error). Anyway, keeping backward 
compatibility for paging state is needed anyway so keeping this ticket to deal 
with that.


> 3.0 paging states are incompatible with pre-3.0 nodes
> -
>
> Key: CASSANDRA-10254
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10254
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Sylvain Lebresne
> Fix For: 3.0.0 rc1
>
>
> When running the upgrade dtest {{TestCQL.static_columns_with_distinct_test}}, 
> the client appears to be passing in an invalid paging state (no cell name). 
> Instead of catching the problem in validation and returning an error to the 
> client, the select statement tries to run it, causing this exception:
> {noformat}
> ERROR QueryMessage.java:135 - Unexpected error during query
> java.lang.ClassCastException: 
> org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be cast 
> to org.apache.cassandra.db.composites.CellName
>   at 
> org.apache.cassandra.service.pager.SliceQueryPager.(SliceQueryPager.java:64)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.pager.MultiPartitionPager.makePager(MultiPartitionPager.java:93)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.pager.MultiPartitionPager.(MultiPartitionPager.java:75)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.pager.QueryPagers.pager(QueryPagers.java:102) 
> ~[main/:na]
>   at 
> org.apache.cassandra.service.pager.QueryPagers.pager(QueryPagers.java:126) 
> ~[main/:na]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:228)
>  ~[main/:na]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:67)
>  ~[main/:na]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:238)
>  ~[main/:na]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) 
> ~[main/:na]
>   at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122)
>  ~[main/:na]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
>  [main/:na]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
>  [main/:na]
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  [netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_40]
>   at 
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>  [main/:na]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [main/:na]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
> {noformat}
> Running {code} UPGRADE_MODE=none nosetests 
> upgrade_tests/cql_tests.py:TestCQL.static_columns_with_distinct_test{code} 
> should cause the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9966) Option to apply statements within a batch sequentially

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747254#comment-14747254
 ] 

Sylvain Lebresne commented on CASSANDRA-9966:
-

Fyi, it's likely just me being dumb, but I've read twice that previous comment 
but have not clue what you are suggesting concretely, nor what you are "not 
super happy" with exactly. Some extra precision/clarification would be much 
appreciated, at least by me.


> Option to apply statements within a batch sequentially
> --
>
> Key: CASSANDRA-9966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9966
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Sam Overton
> Fix For: 3.x, 2.2.x
>
>
> It is possible to batch CAS statements such that their outcome is different 
> to the outcome were they executed sequentially outside of a batch.
> eg.
> a | b | c
> a | 1 | 1
> BEGIN BATCH
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> APPLY BATCH
> results in 
> a | b | c
> a | 2 | 2
> If these statements were not batched, the outcome would be 
> UPDATE foo SET b=2 WHERE a='a' iF c=1
> a | b | c
> a | 2 | 1
> UPDATE foo SET c=2 WHERE a='a' IF b=1
> applied=false (pre-condition b=1 not met)
> Cassandra already checks for incompatible preconditions within a batch (eg 
> one statement with IF c=1 and another statement with IF c=2). It should also 
> check for mutations to columns in one statement that affect the 
> pre-conditions of another statement, or it should evaluate the statement 
> pre-conditions sequentially after applying the mutations of the previous 
> statement to an in-memory model of the partition.
> For backwards compatibility this would have to be a new "strict" batch mode, 
> eg.
> BEGIN STRICT BATCH



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747264#comment-14747264
 ] 

Sylvain Lebresne commented on CASSANDRA-10351:
--

And of course, I mean "+1 but we should probably wait on CI results to be sure 
nothing has been forgotten".

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

[aleksey]

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4b1d59e1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4b1d59e1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4b1d59e1

Branch: refs/heads/cassandra-2.1
Commit: 4b1d59e13143f5a55478a7cfcaca61ba22259f9f
Parents: b8b4eb7 257cdaa
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:44:28 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:44:28 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 7 files changed, 333 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/CHANGES.txt
--
diff --cc CHANGES.txt
index 5f11049,3c47427..12523be
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,46 -1,7 +1,47 @@@
 -2.0.17
 +2.1.10
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * Added configurable warning threshold for GC duration (CASSANDRA-8907)
 + * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 + * Fix handling of streaming EOF (CASSANDRA-10206)
 + * Only check KeyCache when it is enabled
 + * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
 + * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 +Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
 - * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
 +
 +
 +2.1.9
 + * Avoid race condition during read repair (CASSANDRA-9460)
 + * (cqlsh) default load-from-file encoding to utf-8 (CASSANDRA-9898)
 + * Avoid returning Permission.NONE when failing to query users table 
(CASSANDRA-10168)
 + * (cqlsh) Allow encoding to be set through command line (CASSANDRA-10004)
 + * Add new JMX methods to change local compaction strategy (CASSANDRA-9965)
 + * Write hints for paxos commits (CASSANDRA-7342)
 + * (cqlsh) Fix timestamps before 1970 on Windows, always
 +   use UTC for timestamp display (CASSANDRA-1)
 + * (cqlsh) Avoid overwriting new config file with old config
 +   when both exist (CASSANDRA-9777)
 + * Release snapshot selfRef when doing snapshot repair (CASSANDRA-9998)
 + * Cannot replace token does not exist - DN node removed as Fat Client 
(CASSANDRA-9871)
 + * Fix handling of enable/disable autocompaction (CASSANDRA-9899)
 + * Commit log segment recycling is disabled by default (CASSANDRA-9896)
 + * Add consistency level to tracing ouput (CASSANDRA-9827)
 + * Fix MarshalException when upgrading superColumn family (CASSANDRA-9582)
 + * Fix broken logging for "empty" flushes in Memtable (CASSANDRA-9837)
 + * Handle corrupt files on startup (CASSANDRA-9686)
 + * Fix clientutil jar and tests (CASSANDRA-9760)
 + * (cqlsh) Allow the SSL protocol version to be specified through the
 +   config file or environment variables (CASSANDRA-9544)
 + * Remove repair snapshot leftover on startup (CASSANDRA-7357)
 + * Use random nodes for batch log when only 2 racks (CASSANDRA-8735)
 + * Ensure atomicity inside thrift and stream session (CASSANDRA-7757)
 + * Fix nodetool info error when the node is not joined (CASSANDRA-9031)
 +Merged from 2.0:
   * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)
   * Add tool to find why expired sstables are not getting dropped 
(CASSANDRA-10015)
   * Remove erroneous pending HH tasks from tpstats/jmx (CASSANDRA-9129)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
index 2c0980a,dd1637d..da43600
--- a/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
@@@ -72,24 -62,7 +72,25 @@@ public class GossipingPropertyFileSnitc
  }
  catch (ConfigurationException e)
 

[jira] [Created] (CASSANDRA-10352) Paging with DISTINCT and IN can throw ClassCastException

[Sylvain Lebresne (JIRA)]

Sylvain Lebresne created CASSANDRA-10352:


 Summary: Paging with DISTINCT and IN can throw ClassCastException
 Key: CASSANDRA-10352
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10352
 Project: Cassandra
  Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 2.1.x


The ctor of {{SliceQueryPager}} has the following code:
{noformat}
if (state != null)
{
// The only case where this could be a non-CellName Composite is if 
it's Composites.EMPTY, but that's not
// valid for PagingState.cellName, so we can safely cast to 
CellName.
lastReturned = (CellName) 
cfm.comparator.fromByteBuffer(state.cellName);
restoreState(state.remaining, true);
}
{noformat}
The assumption that we can only get a {{CellName}} is invalid however. When 
using a {{MultiPartitionPager}} (so when the query has a {{IN}} on the 
partition key), it's possible for a page to stop at the end of one of the 
underlying pager, and for that pager to be exhausted. If that's the case, the 
{{PagingState}} returned will have the partition key of the next key to 
retrieve, but a {{null}} cellname (since we haven't started to query that next 
key). When the next page is queried, that {{PagingState}} will trigger a 
{{ClassCastException}}.

The [following 
dtest|https://github.com/pcmanus/cassandra-dtest/commit/b98a112fc22f92aa965d2068345c708ae43b8a85]
 reproduces this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10351) We should use unsignedVInt, instead of VInt methods, for size serializations

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747261#comment-14747261
 ] 

Sylvain Lebresne commented on CASSANDRA-10351:
--

bq. got a follow up commit that also reintroduces a message to 
writeUnsignedVInt, if you think it adds informational value.

Certainly doesn't hurt.

+1, thanks.

> We should use unsignedVInt, instead of VInt methods, for size serializations
> 
>
> Key: CASSANDRA-10351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10351
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.0 rc1
>
>
> It's not a huge deal, but it means we more rapidly use an extra byte, and 
> just as importantly we also have more work to do to interpret the data. Since 
> we do this a lot, we may as well keep it as efficient as possible for scalar 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5b05b682
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5b05b682
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5b05b682

Branch: refs/heads/cassandra-3.0
Commit: 5b05b682641f634d8b6548d225b9130b771df969
Parents: e7c2952 4b1d59e
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:48:56 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:48:56 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/CHANGES.txt
--
diff --cc CHANGES.txt
index b0ade42,12523be..7b5fa78
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -20,8 -10,8 +20,9 @@@ Merged from 2.1
   * Only check KeyCache when it is enabled
   * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
   * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 + * Add nodetool gettraceprobability command (CASSANDRA-10234)
  Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
  
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/TokenMetadata.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/service/StorageService.java
--

[2/5] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

[aleksey]

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4b1d59e1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4b1d59e1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4b1d59e1

Branch: refs/heads/trunk
Commit: 4b1d59e13143f5a55478a7cfcaca61ba22259f9f
Parents: b8b4eb7 257cdaa
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:44:28 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:44:28 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 7 files changed, 333 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/CHANGES.txt
--
diff --cc CHANGES.txt
index 5f11049,3c47427..12523be
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,46 -1,7 +1,47 @@@
 -2.0.17
 +2.1.10
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * Added configurable warning threshold for GC duration (CASSANDRA-8907)
 + * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 + * Fix handling of streaming EOF (CASSANDRA-10206)
 + * Only check KeyCache when it is enabled
 + * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
 + * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 +Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
 - * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
 +
 +
 +2.1.9
 + * Avoid race condition during read repair (CASSANDRA-9460)
 + * (cqlsh) default load-from-file encoding to utf-8 (CASSANDRA-9898)
 + * Avoid returning Permission.NONE when failing to query users table 
(CASSANDRA-10168)
 + * (cqlsh) Allow encoding to be set through command line (CASSANDRA-10004)
 + * Add new JMX methods to change local compaction strategy (CASSANDRA-9965)
 + * Write hints for paxos commits (CASSANDRA-7342)
 + * (cqlsh) Fix timestamps before 1970 on Windows, always
 +   use UTC for timestamp display (CASSANDRA-1)
 + * (cqlsh) Avoid overwriting new config file with old config
 +   when both exist (CASSANDRA-9777)
 + * Release snapshot selfRef when doing snapshot repair (CASSANDRA-9998)
 + * Cannot replace token does not exist - DN node removed as Fat Client 
(CASSANDRA-9871)
 + * Fix handling of enable/disable autocompaction (CASSANDRA-9899)
 + * Commit log segment recycling is disabled by default (CASSANDRA-9896)
 + * Add consistency level to tracing ouput (CASSANDRA-9827)
 + * Fix MarshalException when upgrading superColumn family (CASSANDRA-9582)
 + * Fix broken logging for "empty" flushes in Memtable (CASSANDRA-9837)
 + * Handle corrupt files on startup (CASSANDRA-9686)
 + * Fix clientutil jar and tests (CASSANDRA-9760)
 + * (cqlsh) Allow the SSL protocol version to be specified through the
 +   config file or environment variables (CASSANDRA-9544)
 + * Remove repair snapshot leftover on startup (CASSANDRA-7357)
 + * Use random nodes for batch log when only 2 racks (CASSANDRA-8735)
 + * Ensure atomicity inside thrift and stream session (CASSANDRA-7757)
 + * Fix nodetool info error when the node is not joined (CASSANDRA-9031)
 +Merged from 2.0:
   * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)
   * Add tool to find why expired sstables are not getting dropped 
(CASSANDRA-10015)
   * Remove erroneous pending HH tasks from tpstats/jmx (CASSANDRA-9129)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
index 2c0980a,dd1637d..da43600
--- a/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
@@@ -72,24 -62,7 +72,25 @@@ public class GossipingPropertyFileSnitc
  }
  catch (ConfigurationException e)
 

[4/5] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aleksey]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8134f09c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8134f09c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8134f09c

Branch: refs/heads/trunk
Commit: 8134f09c2bdf2d859135a187a9de91f4cfb7620d
Parents: c694cd6 5b05b68
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 12:04:53 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 12:04:53 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 ++---
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 213 ++-
 6 files changed, 335 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --cc src/java/org/apache/cassandra/locator/TokenMetadata.java
index c3ec353,8d54ead..75448a8
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@@ -124,18 -119,9 +124,18 @@@ public class TokenMetadat
  sortedTokens = sortTokens();
  }
  
 +/**
 + * To be used by tests only (via {@link 
StorageService.setPartitionerUnsafe}).
 + */
 +@VisibleForTesting
 +public TokenMetadata cloneWithNewPartitioner(IPartitioner newPartitioner)
 +{
 +return new TokenMetadata(tokenToEndpointMap, endpointToHostIdMap, 
topology, newPartitioner);
 +}
 +
  private ArrayList sortTokens()
  {
- return new ArrayList(tokenToEndpointMap.keySet());
+ return new ArrayList<>(tokenToEndpointMap.keySet());
  }
  
  /** @return the number of nodes bootstrapping into source's primary range 
*/
@@@ -533,10 -551,9 +565,10 @@@
  lock.readLock().lock();
  try
  {
- return new TokenMetadata(SortedBiMultiValMap.create(tokenToEndpointMap, null, inetaddressCmp),
+ return new 
TokenMetadata(SortedBiMultiValMap.create(tokenToEndpointMap, null, 
inetaddressCmp),
   HashBiMap.create(endpointToHostIdMap),
 - new Topology(topology));
 + new Topology(topology),
 + partitioner);
  }
  finally
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/src/java/org/apache/cassandra/service/StorageService.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8134f09c/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
--
diff --cc test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
index 68d0dfa,fc8095d..e7bb70a
--- a/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
+++ b/test/unit/org/apache/cassandra/locator/TokenMetadataTest.java
@@@ -29,9 -42,6 +41,7 @@@ import org.apache.cassandra.config.Data
  import org.apache.cassandra.dht.Token;
  import org.apache.cassandra.service.StorageService;
  
- import static org.apache.cassandra.Util.token;
- import static org.junit.Assert.assertEquals;
 +
  @RunWith(OrderedJUnit4ClassRunner.class)
  public class TokenMetadataTest
  {

[1/5] cassandra git commit: Fix consolidating racks violating the RF contract

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 48da81dc4 -> f3ff8df05


Fix consolidating racks violating the RF contract

patch by Stefania Alborghetti; reviewed by Blake Eggleston for
CASSANDRA-10238


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/257cdaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/257cdaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/257cdaa0

Branch: refs/heads/trunk
Commit: 257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b
Parents: 0bb32f0
Author: Stefania Alborghetti 
Authored: Fri Sep 11 16:31:40 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:40:31 2015 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d4cc15f..3c47427 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.17
+ * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
  * Disallow decommission when node is in drained state (CASSANDRA-8741)
  * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
  * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java 
b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
index 4f822c6..745eeb8 100644
--- a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
@@ -69,6 +69,7 @@ public class PropertyFileSnitch extends 
AbstractNetworkTopologySnitch
 protected void runMayThrow() throws ConfigurationException
 {
 reloadConfiguration();
+StorageService.instance.updateTopology();
 }
 };
 ResourceWatcher.watch(SNITCH_PROPERTIES_FILENAME, runnable, 60 * 
1000);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java 
b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index a673c94..b1b25e8 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -78,14 +78,14 @@ public class TokenMetadata
 // Finally, note that recording the tokens of joining nodes in 
bootstrapTokens also
 // means we can detect and reject the addition of multiple nodes at the 
same token
 // before one becomes part of the ring.
-private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap();
+private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap<>();
 // (don't need to record Token here since it's still part of 
tokenToEndpointMap until it's done leaving)
-private final Set leavingEndpoints = new 
HashSet();
+private final Set leavingEndpoints = new HashSet<>();
 // this is a cache of the calculation from {tokenToEndpointMap, 
bootstrapTokens, leavingEndpoints}
-private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap>();
+private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap<>();
 
 // nodes which are migrating to the new tokens in the ring
-private final Set> movingEndpoints = new 
HashSet>();
+private final Set> movingEndpoints = new 
HashSet<>();
 
 /* Use this lock for manipulating the token map */
 private final ReadWriteLock lock = new ReentrantReadWriteLock(true);
@@ -121,7 +121,7 @@ public class TokenMetadata
 
 private ArrayList sortTokens()
 {
-return new ArrayList(tokenToEndpointMap.keySet());
+return new ArrayList<>(tokenToEndpointMap.keySet());
 }
 
 /** @return the 

[3/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5b05b682
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5b05b682
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5b05b682

Branch: refs/heads/trunk
Commit: 5b05b682641f634d8b6548d225b9130b771df969
Parents: e7c2952 4b1d59e
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:48:56 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:48:56 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/CHANGES.txt
--
diff --cc CHANGES.txt
index b0ade42,12523be..7b5fa78
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -20,8 -10,8 +20,9 @@@ Merged from 2.1
   * Only check KeyCache when it is enabled
   * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
   * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 + * Add nodetool gettraceprobability command (CASSANDRA-10234)
  Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
  
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/locator/TokenMetadata.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b05b682/src/java/org/apache/cassandra/service/StorageService.java
--

[1/4] cassandra git commit: Fix consolidating racks violating the RF contract

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 c694cd62a -> 8134f09c2


Fix consolidating racks violating the RF contract

patch by Stefania Alborghetti; reviewed by Blake Eggleston for
CASSANDRA-10238


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/257cdaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/257cdaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/257cdaa0

Branch: refs/heads/cassandra-3.0
Commit: 257cdaa08dc12f747a25c03b9b0ad3ffc76ace9b
Parents: 0bb32f0
Author: Stefania Alborghetti 
Authored: Fri Sep 11 16:31:40 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:40:31 2015 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 6 files changed, 332 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d4cc15f..3c47427 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.17
+ * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
  * Disallow decommission when node is in drained state (CASSANDRA-8741)
  * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
  * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java 
b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
index 4f822c6..745eeb8 100644
--- a/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
@@ -69,6 +69,7 @@ public class PropertyFileSnitch extends 
AbstractNetworkTopologySnitch
 protected void runMayThrow() throws ConfigurationException
 {
 reloadConfiguration();
+StorageService.instance.updateTopology();
 }
 };
 ResourceWatcher.watch(SNITCH_PROPERTIES_FILENAME, runnable, 60 * 
1000);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/257cdaa0/src/java/org/apache/cassandra/locator/TokenMetadata.java
--
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java 
b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index a673c94..b1b25e8 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -78,14 +78,14 @@ public class TokenMetadata
 // Finally, note that recording the tokens of joining nodes in 
bootstrapTokens also
 // means we can detect and reject the addition of multiple nodes at the 
same token
 // before one becomes part of the ring.
-private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap();
+private final BiMultiValMap bootstrapTokens = new 
BiMultiValMap<>();
 // (don't need to record Token here since it's still part of 
tokenToEndpointMap until it's done leaving)
-private final Set leavingEndpoints = new 
HashSet();
+private final Set leavingEndpoints = new HashSet<>();
 // this is a cache of the calculation from {tokenToEndpointMap, 
bootstrapTokens, leavingEndpoints}
-private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap>();
+private final ConcurrentMap> 
pendingRanges = new ConcurrentHashMap<>();
 
 // nodes which are migrating to the new tokens in the ring
-private final Set> movingEndpoints = new 
HashSet>();
+private final Set> movingEndpoints = new 
HashSet<>();
 
 /* Use this lock for manipulating the token map */
 private final ReadWriteLock lock = new ReentrantReadWriteLock(true);
@@ -121,7 +121,7 @@ public class TokenMetadata
 
 private ArrayList sortTokens()
 {
-return new ArrayList(tokenToEndpointMap.keySet());
+return new ArrayList<>(tokenToEndpointMap.keySet());
 }
 
 

[2/4] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

[aleksey]

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4b1d59e1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4b1d59e1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4b1d59e1

Branch: refs/heads/cassandra-3.0
Commit: 4b1d59e13143f5a55478a7cfcaca61ba22259f9f
Parents: b8b4eb7 257cdaa
Author: Aleksey Yeschenko 
Authored: Wed Sep 16 11:44:28 2015 +0100
Committer: Aleksey Yeschenko 
Committed: Wed Sep 16 11:44:28 2015 +0100

--
 CHANGES.txt |   1 +
 .../locator/GossipingPropertyFileSnitch.java|   1 +
 .../cassandra/locator/PropertyFileSnitch.java   |   1 +
 .../apache/cassandra/locator/TokenMetadata.java | 142 +
 .../locator/YamlFileNetworkTopologySnitch.java  |   1 +
 .../cassandra/service/StorageService.java   |  16 ++
 .../cassandra/locator/TokenMetadataTest.java| 209 ++-
 7 files changed, 333 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/CHANGES.txt
--
diff --cc CHANGES.txt
index 5f11049,3c47427..12523be
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,46 -1,7 +1,47 @@@
 -2.0.17
 +2.1.10
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * Added configurable warning threshold for GC duration (CASSANDRA-8907)
 + * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 + * Fix handling of streaming EOF (CASSANDRA-10206)
 + * Only check KeyCache when it is enabled
 + * Change streaming_socket_timeout_in_ms default to 1 hour (CASSANDRA-8611)
 + * (cqlsh) update list of CQL keywords (CASSANDRA-9232)
 +Merged from 2.0:
+  * Fix consolidating racks violating the RF contract (CASSANDRA-10238)
   * Disallow decommission when node is in drained state (CASSANDRA-8741)
 - * Backport CASSANDRA-8013 to 2.0 (CASSANDRA-10144)
 +
 +
 +2.1.9
 + * Avoid race condition during read repair (CASSANDRA-9460)
 + * (cqlsh) default load-from-file encoding to utf-8 (CASSANDRA-9898)
 + * Avoid returning Permission.NONE when failing to query users table 
(CASSANDRA-10168)
 + * (cqlsh) Allow encoding to be set through command line (CASSANDRA-10004)
 + * Add new JMX methods to change local compaction strategy (CASSANDRA-9965)
 + * Write hints for paxos commits (CASSANDRA-7342)
 + * (cqlsh) Fix timestamps before 1970 on Windows, always
 +   use UTC for timestamp display (CASSANDRA-1)
 + * (cqlsh) Avoid overwriting new config file with old config
 +   when both exist (CASSANDRA-9777)
 + * Release snapshot selfRef when doing snapshot repair (CASSANDRA-9998)
 + * Cannot replace token does not exist - DN node removed as Fat Client 
(CASSANDRA-9871)
 + * Fix handling of enable/disable autocompaction (CASSANDRA-9899)
 + * Commit log segment recycling is disabled by default (CASSANDRA-9896)
 + * Add consistency level to tracing ouput (CASSANDRA-9827)
 + * Fix MarshalException when upgrading superColumn family (CASSANDRA-9582)
 + * Fix broken logging for "empty" flushes in Memtable (CASSANDRA-9837)
 + * Handle corrupt files on startup (CASSANDRA-9686)
 + * Fix clientutil jar and tests (CASSANDRA-9760)
 + * (cqlsh) Allow the SSL protocol version to be specified through the
 +   config file or environment variables (CASSANDRA-9544)
 + * Remove repair snapshot leftover on startup (CASSANDRA-7357)
 + * Use random nodes for batch log when only 2 racks (CASSANDRA-8735)
 + * Ensure atomicity inside thrift and stream session (CASSANDRA-7757)
 + * Fix nodetool info error when the node is not joined (CASSANDRA-9031)
 +Merged from 2.0:
   * Make getFullyExpiredSSTables less expensive (CASSANDRA-9882)
   * Add tool to find why expired sstables are not getting dropped 
(CASSANDRA-10015)
   * Remove erroneous pending HH tasks from tpstats/jmx (CASSANDRA-9129)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4b1d59e1/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
index 2c0980a,dd1637d..da43600
--- a/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
+++ b/src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
@@@ -72,24 -62,7 +72,25 @@@ public class GossipingPropertyFileSnitc
  }
  catch (ConfigurationException e)
 

[jira] [Commented] (CASSANDRA-9839) Move crc_check_chance out of compressions options

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747305#comment-14747305
 ] 

Paulo Motta commented on CASSANDRA-9839:


previous issue was fixed, rebased against 3.0 (tests are running)

> Move crc_check_chance out of compressions options
> -
>
> Key: CASSANDRA-9839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9839
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: client-impacting, docs-impacting
> Fix For: 3.0.0 rc1
>
>
> Follow up to CASSANDRA-8384. The option doesn't belong to compression params 
> - it doesn't affect compression, itself, and isn't passed to compressors upon 
> initialization.
> While it's true that it is (currently) only being honored when reading 
> compressed sstables, it still doesn't belong to compression params (and is 
> causing CASSANDRA-7978 -like issues).
> [~tjake] suggested we should make it an option of its own, and I think we 
> should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-10353) NULL in compaction_history in 2.2.1

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita resolved CASSANDRA-10353.

Resolution: Duplicate

> NULL in compaction_history in 2.2.1
> ---
>
> Key: CASSANDRA-10353
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10353
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Tommy Stendahl
>
> While doing some tests on 2.2.1 I found a problem with a null value in the 
> compaction_history table that results in a nullpointer exception when doing 
> 'nodetool compactionhistory'.
> {quote}
> $ ccm node1 nodetool compactionhistory
> Compaction History:
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at com.google.common.base.Joiner$MapJoiner.join(Joiner.java:330)
> at org.apache.cassandra.utils.FBUtilities.toString(FBUtilities.java:477)
> at 
> org.apache.cassandra.db.compaction.CompactionHistoryTabularData.from(CompactionHistoryTabularData.java:78)
> at 
> org.apache.cassandra.db.SystemKeyspace.getCompactionHistory(SystemKeyspace.java:425)
> at 
> org.apache.cassandra.db.compaction.CompactionManager.getCompactionHistory(CompactionManager.java:1492)
> {quote}
> The problem is the null value in the rows_merged column.
> {quote}
> cqlsh> SELECT * FROM system.compaction_history ;
>  id   | bytes_in | bytes_out | 
> columnfamily_name   | compacted_at | keyspace_name | 
> rows_merged
> --+--+---+-+--+---+
>  86548100-5adf-11e5-a4f4-29d398af617a |  366 |64 | 
> compactions_in_progress | 2015-09-14 12:52:58+ |system |  
>\{2: 2}
>  9ff24610-5adf-11e5-a4f4-29d398af617a |   30 | 0 |
>  bar | 2015-09-14 12:53:41+ |   foo |   null
>  e4ffb130-5ade-11e5-a4f4-29d398af617a |  366 |64 | 
> compactions_in_progress | 2015-09-14 12:48:27+ |system |  
>\{2: 2}
> {quote}
> After some more testing and digging in the code I realised that this is the 
> result of a compaction where all sstables are dropped (i.e. no new sstable is 
> created).
> I tried to recreate this using 2.1.9 and found that there where nothing 
> inserted into the compaction_history table if the same thing happens. That is 
>  due to a check in CompactionTask.runMayThrow() line 181.
> {code}
> if (!iter.hasNext())
> {
> // don't mark compacted in the finally block, since if there _is_ 
> nondeleted data,
> // we need to sync it (via closeAndOpen) first, so there is no period 
> during which
> // a crash could cause data loss.
> cfs.markObsolete(sstables, compactionType);
> return;
> }
> {code}
> The result of the return here is that we end the compaction without inserting 
> anything in the compaction_history table, 2.2.1 does not have this logic and 
> will insert the null value.
> Not sure what the best solution would be:
> 1. Implement the same behaviour as in 2.1.9 and skip inserting anything into  
> compaction_history.
> 2. Make sure we insert a more sensible value then null, but I'm not sure what 
> that value would be.
> 3. Have nodetool handle the null value without a nullpointer exception. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Default coordinator batchlog off with option to enable

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 649a106c3 -> a22d16b7a


Default coordinator batchlog off with option to enable

Patch by tjake; reviewed by Joel Knighton for CASSANDRA-10230


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a22d16b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a22d16b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a22d16b7

Branch: refs/heads/cassandra-3.0
Commit: a22d16b7a347ea43631d343eec47209904bd42d3
Parents: 649a106
Author: T Jake Luciani 
Authored: Tue Sep 15 14:22:35 2015 -0400
Committer: T Jake Luciani 
Committed: Wed Sep 16 09:30:11 2015 -0400

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/db/view/MaterializedViewManager.java| 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22d16b7/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4b2ce1e..7bb5b71 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.0-rc1
+ * Add flag to enable/disable coordinator batchlog for MV writes 
(CASSANDRA-10230)
  * Update cqlsh COPY for new internal driver serialization interface 
(CASSANDRA-10318)
  * Give index implementations more control over rebuild operations 
(CASSANDRA-10312)
  * Update index file format (CASSANDRA-10314)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22d16b7/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
--
diff --git a/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java 
b/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
index ac6a256..41f4ed0 100644
--- a/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
+++ b/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
@@ -59,7 +59,7 @@ import org.apache.cassandra.service.StorageService;
 public class MaterializedViewManager
 {
 private static final Striped LOCKS = 
Striped.lazyWeakLock(DatabaseDescriptor.getConcurrentWriters() * 1024);
-private static final boolean disableCoordinatorBatchlog = 
Boolean.getBoolean("cassandra.mv_disable_coordinator_batchlog");
+private static final boolean enableCoordinatorBatchlog = 
Boolean.getBoolean("cassandra.mv_enable_coordinator_batchlog");
 
 private final ConcurrentNavigableMap viewsByName;
 
@@ -196,7 +196,7 @@ public class MaterializedViewManager
 
 public static boolean updatesAffectView(Collection 
mutations, boolean coordinatorBatchlog)
 {
-if (coordinatorBatchlog && disableCoordinatorBatchlog)
+if (coordinatorBatchlog && !enableCoordinatorBatchlog)
 return false;
 
 for (IMutation mutation : mutations)

[1/2] cassandra git commit: Default coordinator batchlog off with option to enable

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/trunk f3ec5b456 -> e2eb867bd


Default coordinator batchlog off with option to enable

Patch by tjake; reviewed by Joel Knighton for CASSANDRA-10230


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a22d16b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a22d16b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a22d16b7

Branch: refs/heads/trunk
Commit: a22d16b7a347ea43631d343eec47209904bd42d3
Parents: 649a106
Author: T Jake Luciani 
Authored: Tue Sep 15 14:22:35 2015 -0400
Committer: T Jake Luciani 
Committed: Wed Sep 16 09:30:11 2015 -0400

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/db/view/MaterializedViewManager.java| 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22d16b7/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4b2ce1e..7bb5b71 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.0-rc1
+ * Add flag to enable/disable coordinator batchlog for MV writes 
(CASSANDRA-10230)
  * Update cqlsh COPY for new internal driver serialization interface 
(CASSANDRA-10318)
  * Give index implementations more control over rebuild operations 
(CASSANDRA-10312)
  * Update index file format (CASSANDRA-10314)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22d16b7/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
--
diff --git a/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java 
b/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
index ac6a256..41f4ed0 100644
--- a/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
+++ b/src/java/org/apache/cassandra/db/view/MaterializedViewManager.java
@@ -59,7 +59,7 @@ import org.apache.cassandra.service.StorageService;
 public class MaterializedViewManager
 {
 private static final Striped LOCKS = 
Striped.lazyWeakLock(DatabaseDescriptor.getConcurrentWriters() * 1024);
-private static final boolean disableCoordinatorBatchlog = 
Boolean.getBoolean("cassandra.mv_disable_coordinator_batchlog");
+private static final boolean enableCoordinatorBatchlog = 
Boolean.getBoolean("cassandra.mv_enable_coordinator_batchlog");
 
 private final ConcurrentNavigableMap viewsByName;
 
@@ -196,7 +196,7 @@ public class MaterializedViewManager
 
 public static boolean updatesAffectView(Collection 
mutations, boolean coordinatorBatchlog)
 {
-if (coordinatorBatchlog && disableCoordinatorBatchlog)
+if (coordinatorBatchlog && !enableCoordinatorBatchlog)
 return false;
 
 for (IMutation mutation : mutations)

[2/2] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[jake]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e2eb867b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e2eb867b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e2eb867b

Branch: refs/heads/trunk
Commit: e2eb867bdb6686f041cffcc5be2c02eee9397e9e
Parents: f3ec5b4 a22d16b
Author: T Jake Luciani 
Authored: Wed Sep 16 09:33:59 2015 -0400
Committer: T Jake Luciani 
Committed: Wed Sep 16 09:33:59 2015 -0400

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/db/view/MaterializedViewManager.java| 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e2eb867b/CHANGES.txt
--
diff --cc CHANGES.txt
index 2296aa7,7bb5b71..512666e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,8 -1,5 +1,9 @@@
 +3.2
 + * Add transparent data encryption core classes (CASSANDRA-9945)
 +
 +
  3.0.0-rc1
+  * Add flag to enable/disable coordinator batchlog for MV writes 
(CASSANDRA-10230)
   * Update cqlsh COPY for new internal driver serialization interface 
(CASSANDRA-10318)
   * Give index implementations more control over rebuild operations 
(CASSANDRA-10312)
   * Update index file format (CASSANDRA-10314)

[04/10] cassandra git commit: Fix NPE in nodetool compactionhistory

[yukim]

Fix NPE in nodetool compactionhistory

patch by Pierre N. and yukim; reviewed by yukim for CASSANDRA-9758


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/588dc06e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/588dc06e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/588dc06e

Branch: refs/heads/trunk
Commit: 588dc06eb8b5a7678128e9f0c0421fc391b14bce
Parents: 4b1d59e
Author: Yuki Morishita 
Authored: Wed Sep 16 08:36:50 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:36:50 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12523be..2787739 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.10
+ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
  * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
  * BATCH statement is broken in cqlsh (CASSANDRA-10272)
  * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 214f2f5..f866610 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -28,7 +28,6 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
-import java.text.NumberFormat;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
@@ -36,6 +35,9 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.zip.Checksum;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
 import com.google.common.base.Joiner;
 import com.google.common.collect.AbstractIterator;
 import org.apache.cassandra.io.util.FileUtils;
@@ -472,8 +474,18 @@ public class FBUtilities
 return s;
 }
 
-public static String toString(Map map)
+/**
+ * Make straing out of the given {@code Map}.
+ *
+ * @param map Map to make string.
+ * @return String representation of all entries in the map,
+ * where key and value pair is concatenated with ':'.
+ */
+@Nonnull
+public static String toString(@Nullable Map map)
 {
+if (map == null)
+return "";
 Joiner.MapJoiner joiner = Joiner.on(", ").withKeyValueSeparator(":");
 return joiner.join(map);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
--
diff --git a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java 
b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
index adf3763..c82bcc9 100644
--- a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
+++ b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
@@ -27,6 +27,11 @@ import java.nio.charset.StandardCharsets;
 import com.google.common.primitives.Ints;
 import org.junit.Test;
 
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.junit.Assert.assertEquals;
+
 public class FBUtilitiesTest
 {
 @Test
@@ -70,6 +75,20 @@ public class FBUtilitiesTest
 }
 }
 
+@Test
+public void testToString()
+{
+// null turns to empty string
+assertEquals("", FBUtilities.toString(null));
+Map map = new TreeMap<>();
+// empty map turns to empty string
+assertEquals("", FBUtilities.toString(map));
+map.put("aaa", "bbb");
+assertEquals("aaa:bbb", FBUtilities.toString(map));
+map.put("ccc", "ddd");
+assertEquals("aaa:bbb, ccc:ddd", FBUtilities.toString(map));
+}
+
 @Test(expected=CharacterCodingException.class)
 public void testDecode() throws IOException
 {

[jira] [Commented] (CASSANDRA-10342) Read defragmentation can cause unnecessary repairs

[Jim Witschey (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14768987#comment-14768987
 ] 

Jim Witschey commented on CASSANDRA-10342:
--

I'm unlikely to make it to these benchmarks this week, but yes, I can run these.

> Read defragmentation can cause unnecessary repairs
> --
>
> Key: CASSANDRA-10342
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10342
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Olsson
>Assignee: Marcus Eriksson
>Priority: Minor
>
> After applying the fix from CASSANDRA-10299 to the cluster we started having 
> a problem of ~20k small sstables appearing for the table with static data 
> when running incremental repair.
> In the logs there were several messages about flushes for that table, one for 
> each repaired range. The flushed sstables were 0.000kb in size with < 100 ops 
> in each. When checking cfstats there were several writes to that table, even 
> though we were only reading from it and read repair did not repair anything.
> After digging around in the codebase I noticed that defragmentation of data 
> can occur while reading, depending on the query and some other conditions. 
> This causes the read data to be inserted again to have it in a more recent 
> sstable, which can be a problem if that data was repaired using incremental 
> repair. The defragmentation is done in 
> [CollationController.java|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/CollationController.java#L151].
> I guess this wasn't a problem with full repairs since I assume that the 
> digest should be the same even if you have two copies of the same data. But 
> with incremental repair this will most probably cause a mismatch between 
> nodes if that data already was repaired, since the other nodes probably won't 
> have that data in their unrepaired set.
> --
> I can add that the problems on our cluster was probably due to the fact that 
> CASSANDRA-10299 caused the same data to be streamed multiple times and ending 
> up in several sstables. One of the conditions for the defragmentation is that 
> the number of sstables read during a read request have to be more than the 
> minimum number of sstables needed for a compaction(> 4 in our case). So 
> normally I don't think this would cause ~20k sstables to appear, we probably 
> hit an extreme.
> One workaround for this is to use another compaction strategy than STCS(it 
> seems to be the only affected strategy, atleast in 2.1), but the solution 
> might be to either make defragmentation configurable per table or avoid 
> reinserting the data if any of the sstables involved in the read are repaired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[02/10] cassandra git commit: Fix NPE in nodetool compactionhistory

[yukim]

Fix NPE in nodetool compactionhistory

patch by Pierre N. and yukim; reviewed by yukim for CASSANDRA-9758


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/588dc06e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/588dc06e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/588dc06e

Branch: refs/heads/cassandra-2.2
Commit: 588dc06eb8b5a7678128e9f0c0421fc391b14bce
Parents: 4b1d59e
Author: Yuki Morishita 
Authored: Wed Sep 16 08:36:50 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:36:50 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12523be..2787739 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.10
+ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
  * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
  * BATCH statement is broken in cqlsh (CASSANDRA-10272)
  * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 214f2f5..f866610 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -28,7 +28,6 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
-import java.text.NumberFormat;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
@@ -36,6 +35,9 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.zip.Checksum;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
 import com.google.common.base.Joiner;
 import com.google.common.collect.AbstractIterator;
 import org.apache.cassandra.io.util.FileUtils;
@@ -472,8 +474,18 @@ public class FBUtilities
 return s;
 }
 
-public static String toString(Map map)
+/**
+ * Make straing out of the given {@code Map}.
+ *
+ * @param map Map to make string.
+ * @return String representation of all entries in the map,
+ * where key and value pair is concatenated with ':'.
+ */
+@Nonnull
+public static String toString(@Nullable Map map)
 {
+if (map == null)
+return "";
 Joiner.MapJoiner joiner = Joiner.on(", ").withKeyValueSeparator(":");
 return joiner.join(map);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
--
diff --git a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java 
b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
index adf3763..c82bcc9 100644
--- a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
+++ b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
@@ -27,6 +27,11 @@ import java.nio.charset.StandardCharsets;
 import com.google.common.primitives.Ints;
 import org.junit.Test;
 
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.junit.Assert.assertEquals;
+
 public class FBUtilitiesTest
 {
 @Test
@@ -70,6 +75,20 @@ public class FBUtilitiesTest
 }
 }
 
+@Test
+public void testToString()
+{
+// null turns to empty string
+assertEquals("", FBUtilities.toString(null));
+Map map = new TreeMap<>();
+// empty map turns to empty string
+assertEquals("", FBUtilities.toString(map));
+map.put("aaa", "bbb");
+assertEquals("aaa:bbb", FBUtilities.toString(map));
+map.put("ccc", "ddd");
+assertEquals("aaa:bbb, ccc:ddd", FBUtilities.toString(map));
+}
+
 @Test(expected=CharacterCodingException.class)
 public void testDecode() throws IOException
 {

[10/10] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[yukim]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2f82a69e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2f82a69e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2f82a69e

Branch: refs/heads/trunk
Commit: 2f82a69ea270aa1afa886af795e69c006ac6daec
Parents: e2eb867 f542a20
Author: Yuki Morishita 
Authored: Wed Sep 16 08:47:19 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:47:19 2015 -0500

--
 CHANGES.txt   |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java   | 15 ++-
 .../apache/cassandra/utils/FBUtilitiesTest.java   | 18 ++
 3 files changed, 33 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2f82a69e/CHANGES.txt
--

[08/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f542a20e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f542a20e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f542a20e

Branch: refs/heads/cassandra-3.0
Commit: f542a20eaa54db21123001c85f211999180bae14
Parents: a22d16b 426109c
Author: Yuki Morishita 
Authored: Wed Sep 16 08:47:14 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:47:14 2015 -0500

--
 CHANGES.txt   |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java   | 15 ++-
 .../apache/cassandra/utils/FBUtilitiesTest.java   | 18 ++
 3 files changed, 33 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f542a20e/CHANGES.txt
--
diff --cc CHANGES.txt
index 7bb5b71,7deebcf..c688615
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,48 -1,8 +1,49 @@@
 -2.2.2
 +3.0.0-rc1
 + * Add flag to enable/disable coordinator batchlog for MV writes 
(CASSANDRA-10230)
 + * Update cqlsh COPY for new internal driver serialization interface 
(CASSANDRA-10318)
 + * Give index implementations more control over rebuild operations 
(CASSANDRA-10312)
 + * Update index file format (CASSANDRA-10314)
 + * Add "shadowable" row tombstones to deal with mv timestamp issues 
(CASSANDRA-10261)
 + * CFS.loadNewSSTables() broken for pre-3.0 sstables
 + * Cache selected index in read command to reduce lookups (CASSANDRA-10215)
 + * Small optimizations of sstable index serialization (CASSANDRA-10232)
 + * Support for both encrypted and unencrypted native transport connections 
(CASSANDRA-9590)
 +Merged from 2.2:
   * Defer default role manager setup until all nodes are on 2.2+ 
(CASSANDRA-9761)
 + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209)
 +Merged from 2.1:
++ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 +
 +
 +3.0.0-beta2
 + * Fix columns returned by AbstractBtreePartitions (CASSANDRA-10220)
 + * Fix backward compatibility issue due to AbstractBounds serialization bug 
(CASSANDRA-9857)
 + * Fix startup error when upgrading nodes (CASSANDRA-10136)
 + * Base table PRIMARY KEY can be assumed to be NOT NULL in MV creation 
(CASSANDRA-10147)
 + * Improve batchlog write patch (CASSANDRA-9673)
 + * Re-apply MaterializedView updates on commitlog replay (CASSANDRA-10164)
 + * Require AbstractType.isByteOrderComparable declaration in constructor 
(CASSANDRA-9901)
 + * Avoid digest mismatch on upgrade to 3.0 (CASSANDRA-9554)
 + * Fix Materialized View builder when adding multiple MVs (CASSANDRA-10156)
 + * Choose better poolingOptions for protocol v4 in cassandra-stress 
(CASSANDRA-10182)
 + * Fix LWW bug affecting Materialized Views (CASSANDRA-10197)
 + * Ensures frozen sets and maps are always sorted (CASSANDRA-10162)
 + * Don't deadlock when flushing CFS backed custom indexes (CASSANDRA-10181)
 + * Fix double flushing of secondary index tables (CASSANDRA-10180)
 + * Fix incorrect handling of range tombstones in thrift (CASSANDRA-10046)
 + * Only use batchlog when paired materialized view replica is remote 
(CASSANDRA-10061)
 + * Reuse TemporalRow when updating multiple MaterializedViews 
(CASSANDRA-10060)
 + * Validate gc_grace_seconds for batchlog writes and MVs (CASSANDRA-9917)
 + * Fix sstablerepairedset (CASSANDRA-10132)
 +Merged from 2.2:
   * Cancel transaction for sstables we wont redistribute index summary
 for (CASSANDRA-10270)
 - * Handle missing RoleManager in config after upgrade to 2.2 
(CASSANDRA-10209) 
   * Retry snapshot deletion after compaction and gc on Windows 
(CASSANDRA-10222)
   * Fix failure to start with space in directory path on Windows 
(CASSANDRA-10239)
   * Fix repair hang when snapshot failed (CASSANDRA-10057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f542a20e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --cc src/java/org/apache/cassandra/utils/FBUtilities.java
index 6d31d8c,6ef64ef..c4b4193
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@@ -26,10 -28,14 +26,13 @@@ import 

[09/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f542a20e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f542a20e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f542a20e

Branch: refs/heads/trunk
Commit: f542a20eaa54db21123001c85f211999180bae14
Parents: a22d16b 426109c
Author: Yuki Morishita 
Authored: Wed Sep 16 08:47:14 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:47:14 2015 -0500

--
 CHANGES.txt   |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java   | 15 ++-
 .../apache/cassandra/utils/FBUtilitiesTest.java   | 18 ++
 3 files changed, 33 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f542a20e/CHANGES.txt
--
diff --cc CHANGES.txt
index 7bb5b71,7deebcf..c688615
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,48 -1,8 +1,49 @@@
 -2.2.2
 +3.0.0-rc1
 + * Add flag to enable/disable coordinator batchlog for MV writes 
(CASSANDRA-10230)
 + * Update cqlsh COPY for new internal driver serialization interface 
(CASSANDRA-10318)
 + * Give index implementations more control over rebuild operations 
(CASSANDRA-10312)
 + * Update index file format (CASSANDRA-10314)
 + * Add "shadowable" row tombstones to deal with mv timestamp issues 
(CASSANDRA-10261)
 + * CFS.loadNewSSTables() broken for pre-3.0 sstables
 + * Cache selected index in read command to reduce lookups (CASSANDRA-10215)
 + * Small optimizations of sstable index serialization (CASSANDRA-10232)
 + * Support for both encrypted and unencrypted native transport connections 
(CASSANDRA-9590)
 +Merged from 2.2:
   * Defer default role manager setup until all nodes are on 2.2+ 
(CASSANDRA-9761)
 + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209)
 +Merged from 2.1:
++ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
 + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
 + * BATCH statement is broken in cqlsh (CASSANDRA-10272)
 + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066)
 + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282)
 + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations
 +   have completed, regardless of the occurence of exceptions (CASSANDRA-10274)
 +
 +
 +3.0.0-beta2
 + * Fix columns returned by AbstractBtreePartitions (CASSANDRA-10220)
 + * Fix backward compatibility issue due to AbstractBounds serialization bug 
(CASSANDRA-9857)
 + * Fix startup error when upgrading nodes (CASSANDRA-10136)
 + * Base table PRIMARY KEY can be assumed to be NOT NULL in MV creation 
(CASSANDRA-10147)
 + * Improve batchlog write patch (CASSANDRA-9673)
 + * Re-apply MaterializedView updates on commitlog replay (CASSANDRA-10164)
 + * Require AbstractType.isByteOrderComparable declaration in constructor 
(CASSANDRA-9901)
 + * Avoid digest mismatch on upgrade to 3.0 (CASSANDRA-9554)
 + * Fix Materialized View builder when adding multiple MVs (CASSANDRA-10156)
 + * Choose better poolingOptions for protocol v4 in cassandra-stress 
(CASSANDRA-10182)
 + * Fix LWW bug affecting Materialized Views (CASSANDRA-10197)
 + * Ensures frozen sets and maps are always sorted (CASSANDRA-10162)
 + * Don't deadlock when flushing CFS backed custom indexes (CASSANDRA-10181)
 + * Fix double flushing of secondary index tables (CASSANDRA-10180)
 + * Fix incorrect handling of range tombstones in thrift (CASSANDRA-10046)
 + * Only use batchlog when paired materialized view replica is remote 
(CASSANDRA-10061)
 + * Reuse TemporalRow when updating multiple MaterializedViews 
(CASSANDRA-10060)
 + * Validate gc_grace_seconds for batchlog writes and MVs (CASSANDRA-9917)
 + * Fix sstablerepairedset (CASSANDRA-10132)
 +Merged from 2.2:
   * Cancel transaction for sstables we wont redistribute index summary
 for (CASSANDRA-10270)
 - * Handle missing RoleManager in config after upgrade to 2.2 
(CASSANDRA-10209) 
   * Retry snapshot deletion after compaction and gc on Windows 
(CASSANDRA-10222)
   * Fix failure to start with space in directory path on Windows 
(CASSANDRA-10239)
   * Fix repair hang when snapshot failed (CASSANDRA-10057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f542a20e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --cc src/java/org/apache/cassandra/utils/FBUtilities.java
index 6d31d8c,6ef64ef..c4b4193
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@@ -26,10 -28,14 +26,13 @@@ import java.security.MessageDigest
  

[03/10] cassandra git commit: Fix NPE in nodetool compactionhistory

[yukim]

Fix NPE in nodetool compactionhistory

patch by Pierre N. and yukim; reviewed by yukim for CASSANDRA-9758


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/588dc06e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/588dc06e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/588dc06e

Branch: refs/heads/cassandra-3.0
Commit: 588dc06eb8b5a7678128e9f0c0421fc391b14bce
Parents: 4b1d59e
Author: Yuki Morishita 
Authored: Wed Sep 16 08:36:50 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:36:50 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12523be..2787739 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.10
+ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
  * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
  * BATCH statement is broken in cqlsh (CASSANDRA-10272)
  * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 214f2f5..f866610 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -28,7 +28,6 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
-import java.text.NumberFormat;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
@@ -36,6 +35,9 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.zip.Checksum;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
 import com.google.common.base.Joiner;
 import com.google.common.collect.AbstractIterator;
 import org.apache.cassandra.io.util.FileUtils;
@@ -472,8 +474,18 @@ public class FBUtilities
 return s;
 }
 
-public static String toString(Map map)
+/**
+ * Make straing out of the given {@code Map}.
+ *
+ * @param map Map to make string.
+ * @return String representation of all entries in the map,
+ * where key and value pair is concatenated with ':'.
+ */
+@Nonnull
+public static String toString(@Nullable Map map)
 {
+if (map == null)
+return "";
 Joiner.MapJoiner joiner = Joiner.on(", ").withKeyValueSeparator(":");
 return joiner.join(map);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
--
diff --git a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java 
b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
index adf3763..c82bcc9 100644
--- a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
+++ b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
@@ -27,6 +27,11 @@ import java.nio.charset.StandardCharsets;
 import com.google.common.primitives.Ints;
 import org.junit.Test;
 
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.junit.Assert.assertEquals;
+
 public class FBUtilitiesTest
 {
 @Test
@@ -70,6 +75,20 @@ public class FBUtilitiesTest
 }
 }
 
+@Test
+public void testToString()
+{
+// null turns to empty string
+assertEquals("", FBUtilities.toString(null));
+Map map = new TreeMap<>();
+// empty map turns to empty string
+assertEquals("", FBUtilities.toString(map));
+map.put("aaa", "bbb");
+assertEquals("aaa:bbb", FBUtilities.toString(map));
+map.put("ccc", "ddd");
+assertEquals("aaa:bbb, ccc:ddd", FBUtilities.toString(map));
+}
+
 @Test(expected=CharacterCodingException.class)
 public void testDecode() throws IOException
 {

[05/10] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[yukim]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/426109c5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/426109c5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/426109c5

Branch: refs/heads/cassandra-3.0
Commit: 426109c50bf7f264c1965dd77482c161cbec0242
Parents: 5b05b68 588dc06
Author: Yuki Morishita 
Authored: Wed Sep 16 08:39:29 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:39:29 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/CHANGES.txt
--
diff --cc CHANGES.txt
index 7b5fa78,2787739..7deebcf
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,5 +1,15 @@@
 -2.1.10
 +2.2.2
 + * Defer default role manager setup until all nodes are on 2.2+ 
(CASSANDRA-9761)
 + * Cancel transaction for sstables we wont redistribute index summary
 +   for (CASSANDRA-10270)
 + * Handle missing RoleManager in config after upgrade to 2.2 
(CASSANDRA-10209) 
 + * Retry snapshot deletion after compaction and gc on Windows 
(CASSANDRA-10222)
 + * Fix failure to start with space in directory path on Windows 
(CASSANDRA-10239)
 + * Fix repair hang when snapshot failed (CASSANDRA-10057)
 + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks
 +   (CASSANDRA-10199)
 +Merged from 2.1:
+  * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
   * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
   * BATCH statement is broken in cqlsh (CASSANDRA-10272)
   * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --cc src/java/org/apache/cassandra/utils/FBUtilities.java
index b41bdab,f866610..6ef64ef
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@@ -26,14 -28,19 +26,16 @@@ import java.net.*
  import java.nio.ByteBuffer;
  import java.security.MessageDigest;
  import java.security.NoSuchAlgorithmException;
- import java.text.NumberFormat;
  import java.util.*;
 -import java.util.concurrent.ExecutionException;
 -import java.util.concurrent.Future;
 -import java.util.concurrent.TimeUnit;
 -import java.util.concurrent.TimeoutException;
 +import java.util.concurrent.*;
 +import java.util.zip.Adler32;
  import java.util.zip.Checksum;
  
+ import javax.annotation.Nonnull;
+ import javax.annotation.Nullable;
+ 
  import com.google.common.base.Joiner;
  import com.google.common.collect.AbstractIterator;
 -import org.apache.cassandra.io.util.FileUtils;
  import org.apache.commons.lang3.StringUtils;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;

[01/10] cassandra git commit: Fix NPE in nodetool compactionhistory

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 4b1d59e13 -> 588dc06eb
  refs/heads/cassandra-2.2 5b05b6826 -> 426109c50
  refs/heads/cassandra-3.0 a22d16b7a -> f542a20ea
  refs/heads/trunk e2eb867bd -> 2f82a69ea


Fix NPE in nodetool compactionhistory

patch by Pierre N. and yukim; reviewed by yukim for CASSANDRA-9758


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/588dc06e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/588dc06e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/588dc06e

Branch: refs/heads/cassandra-2.1
Commit: 588dc06eb8b5a7678128e9f0c0421fc391b14bce
Parents: 4b1d59e
Author: Yuki Morishita 
Authored: Wed Sep 16 08:36:50 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:36:50 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12523be..2787739 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.10
+ * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
  * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
  * BATCH statement is broken in cqlsh (CASSANDRA-10272)
  * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 214f2f5..f866610 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -28,7 +28,6 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
-import java.text.NumberFormat;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
@@ -36,6 +35,9 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.zip.Checksum;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
 import com.google.common.base.Joiner;
 import com.google.common.collect.AbstractIterator;
 import org.apache.cassandra.io.util.FileUtils;
@@ -472,8 +474,18 @@ public class FBUtilities
 return s;
 }
 
-public static String toString(Map map)
+/**
+ * Make straing out of the given {@code Map}.
+ *
+ * @param map Map to make string.
+ * @return String representation of all entries in the map,
+ * where key and value pair is concatenated with ':'.
+ */
+@Nonnull
+public static String toString(@Nullable Map map)
 {
+if (map == null)
+return "";
 Joiner.MapJoiner joiner = Joiner.on(", ").withKeyValueSeparator(":");
 return joiner.join(map);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/588dc06e/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
--
diff --git a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java 
b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
index adf3763..c82bcc9 100644
--- a/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
+++ b/test/unit/org/apache/cassandra/utils/FBUtilitiesTest.java
@@ -27,6 +27,11 @@ import java.nio.charset.StandardCharsets;
 import com.google.common.primitives.Ints;
 import org.junit.Test;
 
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.junit.Assert.assertEquals;
+
 public class FBUtilitiesTest
 {
 @Test
@@ -70,6 +75,20 @@ public class FBUtilitiesTest
 }
 }
 
+@Test
+public void testToString()
+{
+// null turns to empty string
+assertEquals("", FBUtilities.toString(null));
+Map map = new TreeMap<>();
+// empty map turns to empty string
+assertEquals("", FBUtilities.toString(map));
+map.put("aaa", "bbb");
+assertEquals("aaa:bbb", FBUtilities.toString(map));
+map.put("ccc", "ddd");
+assertEquals("aaa:bbb, ccc:ddd", FBUtilities.toString(map));
+}
+
 @Test(expected=CharacterCodingException.class)
 public void testDecode() throws IOException
 {

[06/10] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[yukim]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/426109c5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/426109c5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/426109c5

Branch: refs/heads/cassandra-2.2
Commit: 426109c50bf7f264c1965dd77482c161cbec0242
Parents: 5b05b68 588dc06
Author: Yuki Morishita 
Authored: Wed Sep 16 08:39:29 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:39:29 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/CHANGES.txt
--
diff --cc CHANGES.txt
index 7b5fa78,2787739..7deebcf
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,5 +1,15 @@@
 -2.1.10
 +2.2.2
 + * Defer default role manager setup until all nodes are on 2.2+ 
(CASSANDRA-9761)
 + * Cancel transaction for sstables we wont redistribute index summary
 +   for (CASSANDRA-10270)
 + * Handle missing RoleManager in config after upgrade to 2.2 
(CASSANDRA-10209) 
 + * Retry snapshot deletion after compaction and gc on Windows 
(CASSANDRA-10222)
 + * Fix failure to start with space in directory path on Windows 
(CASSANDRA-10239)
 + * Fix repair hang when snapshot failed (CASSANDRA-10057)
 + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks
 +   (CASSANDRA-10199)
 +Merged from 2.1:
+  * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
   * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
   * BATCH statement is broken in cqlsh (CASSANDRA-10272)
   * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --cc src/java/org/apache/cassandra/utils/FBUtilities.java
index b41bdab,f866610..6ef64ef
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@@ -26,14 -28,19 +26,16 @@@ import java.net.*
  import java.nio.ByteBuffer;
  import java.security.MessageDigest;
  import java.security.NoSuchAlgorithmException;
- import java.text.NumberFormat;
  import java.util.*;
 -import java.util.concurrent.ExecutionException;
 -import java.util.concurrent.Future;
 -import java.util.concurrent.TimeUnit;
 -import java.util.concurrent.TimeoutException;
 +import java.util.concurrent.*;
 +import java.util.zip.Adler32;
  import java.util.zip.Checksum;
  
+ import javax.annotation.Nonnull;
+ import javax.annotation.Nullable;
+ 
  import com.google.common.base.Joiner;
  import com.google.common.collect.AbstractIterator;
 -import org.apache.cassandra.io.util.FileUtils;
  import org.apache.commons.lang3.StringUtils;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;

[07/10] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[yukim]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/426109c5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/426109c5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/426109c5

Branch: refs/heads/trunk
Commit: 426109c50bf7f264c1965dd77482c161cbec0242
Parents: 5b05b68 588dc06
Author: Yuki Morishita 
Authored: Wed Sep 16 08:39:29 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Sep 16 08:39:29 2015 -0500

--
 CHANGES.txt  |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java  | 16 ++--
 .../apache/cassandra/utils/FBUtilitiesTest.java  | 19 +++
 3 files changed, 34 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/CHANGES.txt
--
diff --cc CHANGES.txt
index 7b5fa78,2787739..7deebcf
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,5 +1,15 @@@
 -2.1.10
 +2.2.2
 + * Defer default role manager setup until all nodes are on 2.2+ 
(CASSANDRA-9761)
 + * Cancel transaction for sstables we wont redistribute index summary
 +   for (CASSANDRA-10270)
 + * Handle missing RoleManager in config after upgrade to 2.2 
(CASSANDRA-10209) 
 + * Retry snapshot deletion after compaction and gc on Windows 
(CASSANDRA-10222)
 + * Fix failure to start with space in directory path on Windows 
(CASSANDRA-10239)
 + * Fix repair hang when snapshot failed (CASSANDRA-10057)
 + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks
 +   (CASSANDRA-10199)
 +Merged from 2.1:
+  * Fix NPE in nodetool compactionhistory (CASSANDRA-9758)
   * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
   * BATCH statement is broken in cqlsh (CASSANDRA-10272)
   * Added configurable warning threshold for GC duration (CASSANDRA-8907)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/426109c5/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --cc src/java/org/apache/cassandra/utils/FBUtilities.java
index b41bdab,f866610..6ef64ef
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@@ -26,14 -28,19 +26,16 @@@ import java.net.*
  import java.nio.ByteBuffer;
  import java.security.MessageDigest;
  import java.security.NoSuchAlgorithmException;
- import java.text.NumberFormat;
  import java.util.*;
 -import java.util.concurrent.ExecutionException;
 -import java.util.concurrent.Future;
 -import java.util.concurrent.TimeUnit;
 -import java.util.concurrent.TimeoutException;
 +import java.util.concurrent.*;
 +import java.util.zip.Adler32;
  import java.util.zip.Checksum;
  
+ import javax.annotation.Nonnull;
+ import javax.annotation.Nullable;
+ 
  import com.google.common.base.Joiner;
  import com.google.common.collect.AbstractIterator;
 -import org.apache.cassandra.io.util.FileUtils;
  import org.apache.commons.lang3.StringUtils;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;

[jira] [Updated] (CASSANDRA-10342) Read defragmentation can cause unnecessary repairs

[Jim Witschey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Witschey updated CASSANDRA-10342:
-
Assignee: Marcus Eriksson
  Tester: Jim Witschey

> Read defragmentation can cause unnecessary repairs
> --
>
> Key: CASSANDRA-10342
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10342
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Olsson
>Assignee: Marcus Eriksson
>Priority: Minor
>
> After applying the fix from CASSANDRA-10299 to the cluster we started having 
> a problem of ~20k small sstables appearing for the table with static data 
> when running incremental repair.
> In the logs there were several messages about flushes for that table, one for 
> each repaired range. The flushed sstables were 0.000kb in size with < 100 ops 
> in each. When checking cfstats there were several writes to that table, even 
> though we were only reading from it and read repair did not repair anything.
> After digging around in the codebase I noticed that defragmentation of data 
> can occur while reading, depending on the query and some other conditions. 
> This causes the read data to be inserted again to have it in a more recent 
> sstable, which can be a problem if that data was repaired using incremental 
> repair. The defragmentation is done in 
> [CollationController.java|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/CollationController.java#L151].
> I guess this wasn't a problem with full repairs since I assume that the 
> digest should be the same even if you have two copies of the same data. But 
> with incremental repair this will most probably cause a mismatch between 
> nodes if that data already was repaired, since the other nodes probably won't 
> have that data in their unrepaired set.
> --
> I can add that the problems on our cluster was probably due to the fact that 
> CASSANDRA-10299 caused the same data to be streamed multiple times and ending 
> up in several sstables. One of the conditions for the defragmentation is that 
> the number of sstables read during a read request have to be more than the 
> minimum number of sstables needed for a compaction(> 4 in our case). So 
> normally I don't think this would cause ~20k sstables to appear, we probably 
> hit an extreme.
> One workaround for this is to use another compaction strategy than STCS(it 
> seems to be the only affected strategy, atleast in 2.1), but the solution 
> might be to either make defragmentation configurable per table or avoid 
> reinserting the data if any of the sstables involved in the read are repaired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)