date:20171206

[jira] [Commented] (CASSANDRA-12503) Structure for netstats output format (JSON, YAML)

2017-12-06 Thread Hiroki Watanabe (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281382#comment-16281382
 ] 

Hiroki Watanabe commented on CASSANDRA-12503:
-

[~multani] [~yukim] 
I'm sorry to have stop developing that long and I have no progress. 
I hope that you take over the patch.

> Structure for netstats output format (JSON, YAML)
> -
>
> Key: CASSANDRA-12503
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12503
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Hiroki Watanabe
>Assignee: Hiroki Watanabe
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: new_receiving.def, new_receiving.json, 
> new_receiving.yaml, new_sending.def, new_sending.json, new_sending.yaml, 
> old_receiving.def, old_sending.def, trunk.patch
>
>
> As with nodetool tpstats and tablestats (CASSANDRA-12035), nodetool netstats 
> should also support useful output formats such as JSON or YAML, so we 
> implemented it. 
> Please review the attached patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13526) nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281371#comment-16281371
 ] 

ASF GitHub Bot commented on CASSANDRA-13526:


Github user asfgit closed the pull request at:

https://github.com/apache/cassandra-dtest/pull/1


> nodetool cleanup on KS with no replicas should remove old data, not silently 
> complete
> -
>
> Key: CASSANDRA-13526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13526
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jeff Jirsa
>Assignee: ZhaoYang
>  Labels: usability
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> From the user list:
> https://lists.apache.org/thread.html/5d49cc6bbc6fd2e5f8b12f2308a3e24212a55afbb441af5cb8cd4167@%3Cuser.cassandra.apache.org%3E
> If you have a multi-dc cluster, but some keyspaces not replicated to a given 
> DC, you'll be unable to run cleanup on those keyspaces in that DC, because 
> [the cleanup code will see no ranges and exit 
> early|https://github.com/apache/cassandra/blob/4cfaf85/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L427-L441]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra-dtest git commit: CASSANDRA-13526: nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread jjirsa

Repository: cassandra-dtest
Updated Branches:
  refs/heads/master ccc6e188b -> 0413754f4


CASSANDRA-13526: nodetool cleanup on KS with no replicas should remove old 
data, not silently complete

Closes #1


Project: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/commit/0413754f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/tree/0413754f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/diff/0413754f

Branch: refs/heads/master
Commit: 0413754f41d5ef94f35d80d91f57c38a80541994
Parents: ccc6e18
Author: Zhao Yang 
Authored: Thu Jul 20 11:18:18 2017 +0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 22:03:53 2017 -0800

--
 nodetool_test.py | 73 ---
 1 file changed, 70 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/0413754f/nodetool_test.py
--
diff --git a/nodetool_test.py b/nodetool_test.py
index 93df2ac..90db848 100644
--- a/nodetool_test.py
+++ b/nodetool_test.py
@@ -1,8 +1,9 @@
 import os
-
+from cassandra import ConsistencyLevel
+from cassandra.query import SimpleStatement
 from ccmlib.node import ToolError
-from dtest import Tester, debug
-from tools.assertions import assert_all, assert_invalid
+from dtest import Tester, debug, create_ks
+from tools.assertions import assert_all, assert_invalid, assert_none
 from tools.decorators import since
 from tools.jmxutils import JolokiaAgent, make_mbean, 
remove_perf_disable_shared_mem
 
@@ -96,6 +97,72 @@ class TestNodetool(Tester):
 debug(out)
 self.assertRegexpMatches(out, r'.* 123 ms')
 
+@since('3.0')
+def test_cleanup_when_no_replica_with_index(self):
+self._cleanup_when_no_replica(True)
+
+@since('3.0')
+def test_cleanup_when_no_replica_without_index(self):
+self._cleanup_when_no_replica(False)
+
+def _cleanup_when_no_replica(self, with_index=False):
+"""
+@jira_ticket CASSANDRA-13526
+Test nodetool cleanup KS to remove old data when new replicas in 
current node instead of directly returning success.
+"""
+self.cluster.populate([1, 1]).start(wait_for_binary_proto=True, 
wait_other_notice=True)
+
+node_dc1 = self.cluster.nodelist()[0]
+node_dc2 = self.cluster.nodelist()[1]
+
+# init schema with rf on both data centers
+replication_factor = {'dc1': 1, 'dc2': 1}
+session = self.patient_exclusive_cql_connection(node_dc1, 
consistency_level=ConsistencyLevel.ALL)
+session_dc2 = self.patient_exclusive_cql_connection(node_dc2, 
consistency_level=ConsistencyLevel.LOCAL_ONE)
+create_ks(session, 'ks', replication_factor)
+session.execute('CREATE TABLE ks.cf (id int PRIMARY KEY, value text) 
with dclocal_read_repair_chance = 0 AND read_repair_chance = 0;', trace=False)
+if with_index:
+session.execute('CREATE INDEX value_by_key on ks.cf(value)', 
trace=False)
+
+# populate data
+for i in range(0, 100):
+session.execute(SimpleStatement("INSERT INTO ks.cf(id, value) 
VALUES({}, 'value');".format(i), consistency_level=ConsistencyLevel.ALL))
+
+# generate sstable
+self.cluster.flush()
+
+for node in self.cluster.nodelist():
+self.assertNotEqual(0, len(node.get_sstables('ks', 'cf')))
+if with_index:
+self.assertEqual(len(list(session_dc2.execute("SELECT * FROM ks.cf 
WHERE value = 'value'"))), 100)
+
+# alter rf to only dc1
+session.execute("ALTER KEYSPACE ks WITH REPLICATION = {'class' : 
'NetworkTopologyStrategy', 'dc1' : 1, 'dc2' : 0};")
+
+# nodetool cleanup on dc2
+node_dc2.nodetool("cleanup ks cf")
+node_dc2.nodetool("compact ks cf")
+
+# check local data on dc2
+for node in self.cluster.nodelist():
+if node.data_center == 'dc2':
+self.assertEqual(0, len(node.get_sstables('ks', 'cf')))
+else:
+self.assertNotEqual(0, len(node.get_sstables('ks', 'cf')))
+
+# dc1 data remains
+statement = SimpleStatement("SELECT * FROM ks.cf", 
consistency_level=ConsistencyLevel.LOCAL_ONE)
+self.assertEqual(len(list(session.execute(statement))), 100)
+if with_index:
+statement = SimpleStatement("SELECT * FROM ks.cf WHERE value = 
'value'", consistency_level=ConsistencyLevel.LOCAL_ONE)
+self.assertEqual(len(list(session.execute(statement))), 100)
+
+# alter rf back to query dc2, no data, no index
+session.execute("ALTER KEYSPACE ks WITH REPLICATION = {'class' : 
'NetworkTopologyStrategy', 'dc1' : 0, 'dc2'

[jira] [Updated] (CASSANDRA-14094) Avoid pointless calls to ThreadLocalRandom

2017-12-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14094:
---
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   4.0
   3.11.2
   3.0.16
   Status: Resolved  (was: Ready to Commit)

Thanks Jason, committed as {{b885e9c0547709152b5a118af30508bf287d3844}} to 3.0 
and merged up to 3.11 and trunk.


> Avoid pointless calls to ThreadLocalRandom
> --
>
> Key: CASSANDRA-14094
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14094
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> In the compression paths, we probabilistically validate the checksum. In 
> cases where the chance is 100%, we don’t need to call {{ThreadLocalRandom}} .



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

2017-12-06 Thread jjirsa

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 090f41883 -> b885e9c05
  refs/heads/cassandra-3.11 c169d491e -> 4e74f0148
  refs/heads/trunk e79e50b0a -> a6f39834b


Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

Patch by Jeff Jirsa; Reviewed by Jason Brown for CASSANDRA-14094


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b885e9c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b885e9c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b885e9c0

Branch: refs/heads/cassandra-3.0
Commit: b885e9c0547709152b5a118af30508bf287d3844
Parents: 090f418
Author: Jeff Jirsa 
Authored: Mon Dec 4 16:32:30 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:55:47 2017 -0800

--
 CHANGES.txt| 1 +
 .../cassandra/io/compress/CompressedRandomAccessReader.java| 6 --
 .../cassandra/streaming/compress/CompressedInputStream.java| 3 ++-
 3 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9638886..b275397 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Optimize CRC check chance probability calculations (CASSANDRA-14094)
  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
--
diff --git 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
index 0624e89..9658316 100644
--- 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
+++ 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
@@ -129,7 +129,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressed.rewind();
 metadata.checksumType.update( checksum, (compressed));
@@ -191,7 +192,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressedChunk.position(chunkOffset).limit(chunkOffset + 
chunk.length);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
--
diff --git 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
index 6577980..e3d698e 100644
--- 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
+++ 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
@@ -136,7 +136,8 @@ public class CompressedInputStream extends InputStream
 totalCompressedBytesRead += compressed.length;
 
 // validate crc randomly
-if (this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
+if (this.crcCheckChanceSupplier.get() >= 1d ||
+this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
 {
 checksum.update(compressed, 0, compressed.length - 
checksumBytes.length);
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e74f014
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e74f014
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e74f014

Branch: refs/heads/trunk
Commit: 4e74f01488e03d85516b68514388c32d3c78965c
Parents: c169d49 b885e9c
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:56:22 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:56:47 2017 -0800

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/io/util/CompressedChunkReader.java| 6 --
 .../cassandra/streaming/compress/CompressedInputStream.java| 3 ++-
 3 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e74f014/CHANGES.txt
--
diff --cc CHANGES.txt
index 3c6565c,b275397..1a1a2cf
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,7 +1,16 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
  3.0.16
+  * Optimize CRC check chance probability calculations (CASSANDRA-14094)
   * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e74f014/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
--
diff --cc src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
index 8f00ce7,000..0919c29
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
+++ b/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
@@@ -1,227 -1,0 +1,229 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +
 +package org.apache.cassandra.io.util;
 +
 +import java.io.IOException;
 +import java.nio.ByteBuffer;
 +import java.util.concurrent.ThreadLocalRandom;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import com.google.common.primitives.Ints;
 +
 +import org.apache.cassandra.io.compress.BufferType;
 +import org.apache.cassandra.io.compress.CompressionMetadata;
 +import org.apache.cassandra.io.compress.CorruptBlockException;
 +import org.apache.cassandra.io.sstable.CorruptSSTableException;
 +
 +public abstract class CompressedChunkReader extends AbstractReaderFileProxy 
implements ChunkReader
 +{
 +final CompressionMetadata metadata;
 +
 +protected CompressedChunkReader(ChannelProxy channel, CompressionMetadata 
metadata)
 +{
 +super(channel, metadata.dataLength);
 +this.metadata = metadata;
 +assert Integer.bitCount(metadata.chunkLength()) == 1; //must be a 
power of two
 +}
 +
 +@VisibleForTesting
 +public double getCrcCheckChance()
 +{
 +return metadata.parameters.getCrcCheckChance();
 +}
 +
 +@Override
 +public String toString()
 +{
 +return String.format("CompressedChunkReader.%s(%s - %s, chunk length 
%d, data length %d)",
 + getClass().getSimpleName(),
 + channel.filePath(),
 +

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6f39834
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6f39834
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6f39834

Branch: refs/heads/trunk
Commit: a6f39834b940afe1d5f09636df754cb50c1240ef
Parents: e79e50b 4e74f01
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:56:56 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:57:20 2017 -0800

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/schema/CompressionParams.java   | 3 ++-
 .../cassandra/streaming/compress/CompressedInputStream.java   | 3 ++-
 3 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6f39834/CHANGES.txt
--
diff --cc CHANGES.txt
index ea4ab0b,1a1a2cf..34af97d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -178,8 -8,9 +178,9 @@@
   * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
  Merged from 3.0:
  3.0.16
+  * Optimize CRC check chance probability calculations (CASSANDRA-14094)
   * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6f39834/src/java/org/apache/cassandra/schema/CompressionParams.java
--
diff --cc src/java/org/apache/cassandra/schema/CompressionParams.java
index 2d60d13,f48a688..b96334b
--- a/src/java/org/apache/cassandra/schema/CompressionParams.java
+++ b/src/java/org/apache/cassandra/schema/CompressionParams.java
@@@ -537,12 -468,6 +537,13 @@@ public final class CompressionParam
  return crcCheckChance;
  }
  
 +public boolean shouldCheckCrc()
 +{
 +double checkChance = getCrcCheckChance();
- return checkChance > 0d && checkChance > 
ThreadLocalRandom.current().nextDouble();
++return checkChance >= 1d ||
++   (checkChance > 0d && checkChance > 
ThreadLocalRandom.current().nextDouble());
 +}
 +
  @Override
  public boolean equals(Object obj)
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6f39834/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
--
diff --cc 
src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
index da63403,8a32d7a..290dd9e
--- 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
+++ 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
@@@ -149,40 -114,63 +149,41 @@@ public class CompressedInputStream exte
  }
  }
  
 -@Override
 -public int read() throws IOException
 -{
 -if (current >= bufferOffset + buffer.length || validBufferBytes == -1)
 -decompressNextChunk();
 -
 -assert current >= bufferOffset && current < bufferOffset + 
validBufferBytes;
 -
 -return ((int) buffer[(int) (current++ - bufferOffset)]) & 0xff;
 -}
 -
 -@Override
 -public int read(byte[] b, int off, int len) throws IOException
 +private void decompress(ByteBuffer compressed) throws IOException
  {
 -long nextCurrent = current + len;
 -
 -if (current >= bufferOffset + buffer.length || validBufferBytes == -1)
 -decompressNextChunk();
 -
 -assert nextCurrent >= bufferOffset;
 +int length = compressed.remaining();
  
 -int read = 0;
 -while (read < len)
 +// uncompress if the buffer size is less than the max chunk size. 
else, if the buffer size is greater than or equal to the maxCompressedLength,
 +// we assume the buffer is not compressed. see CASSANDRA-10520
 +final boolean releaseCompressedBuffer;
 +if (length - CHECKSUM_LENGTH < info.parameters.maxCompressedLength())
  {
 -int nextLen = Math.min((len - read), (int)((bufferOffset + 
validBufferBytes) - current));
 -
 -System.arraycopy(buffer, (int)(current - bufferOffset), b, off + 
read, nextLen);
 -read += nextLen;
 -
 -current += nextLen;
 -if (read != len)
 -decompressNextChunk();
 +buffer.clear();

[3/6] cassandra git commit: Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

2017-12-06 Thread jjirsa

Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

Patch by Jeff Jirsa; Reviewed by Jason Brown for CASSANDRA-14094


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b885e9c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b885e9c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b885e9c0

Branch: refs/heads/trunk
Commit: b885e9c0547709152b5a118af30508bf287d3844
Parents: 090f418
Author: Jeff Jirsa 
Authored: Mon Dec 4 16:32:30 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:55:47 2017 -0800

--
 CHANGES.txt| 1 +
 .../cassandra/io/compress/CompressedRandomAccessReader.java| 6 --
 .../cassandra/streaming/compress/CompressedInputStream.java| 3 ++-
 3 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9638886..b275397 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Optimize CRC check chance probability calculations (CASSANDRA-14094)
  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
--
diff --git 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
index 0624e89..9658316 100644
--- 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
+++ 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
@@ -129,7 +129,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressed.rewind();
 metadata.checksumType.update( checksum, (compressed));
@@ -191,7 +192,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressedChunk.position(chunkOffset).limit(chunkOffset + 
chunk.length);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
--
diff --git 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
index 6577980..e3d698e 100644
--- 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
+++ 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
@@ -136,7 +136,8 @@ public class CompressedInputStream extends InputStream
 totalCompressedBytesRead += compressed.length;
 
 // validate crc randomly
-if (this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
+if (this.crcCheckChanceSupplier.get() >= 1d ||
+this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
 {
 checksum.update(compressed, 0, compressed.length - 
checksumBytes.length);
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

2017-12-06 Thread jjirsa

Avoid pointless calls to ThreadLocalRandom during CRC probability calculations

Patch by Jeff Jirsa; Reviewed by Jason Brown for CASSANDRA-14094


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b885e9c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b885e9c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b885e9c0

Branch: refs/heads/cassandra-3.11
Commit: b885e9c0547709152b5a118af30508bf287d3844
Parents: 090f418
Author: Jeff Jirsa 
Authored: Mon Dec 4 16:32:30 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:55:47 2017 -0800

--
 CHANGES.txt| 1 +
 .../cassandra/io/compress/CompressedRandomAccessReader.java| 6 --
 .../cassandra/streaming/compress/CompressedInputStream.java| 3 ++-
 3 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9638886..b275397 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Optimize CRC check chance probability calculations (CASSANDRA-14094)
  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
--
diff --git 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
index 0624e89..9658316 100644
--- 
a/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
+++ 
b/src/java/org/apache/cassandra/io/compress/CompressedRandomAccessReader.java
@@ -129,7 +129,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressed.rewind();
 metadata.checksumType.update( checksum, (compressed));
@@ -191,7 +192,8 @@ public class CompressedRandomAccessReader extends 
RandomAccessReader
 buffer.flip();
 }
 
-if (getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
+if (getCrcCheckChance() >= 1d ||
+getCrcCheckChance() > ThreadLocalRandom.current().nextDouble())
 {
 compressedChunk.position(chunkOffset).limit(chunkOffset + 
chunk.length);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b885e9c0/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
--
diff --git 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
index 6577980..e3d698e 100644
--- 
a/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
+++ 
b/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java
@@ -136,7 +136,8 @@ public class CompressedInputStream extends InputStream
 totalCompressedBytesRead += compressed.length;
 
 // validate crc randomly
-if (this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
+if (this.crcCheckChanceSupplier.get() >= 1d ||
+this.crcCheckChanceSupplier.get() > 
ThreadLocalRandom.current().nextDouble())
 {
 checksum.update(compressed, 0, compressed.length - 
checksumBytes.length);
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e74f014
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e74f014
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e74f014

Branch: refs/heads/cassandra-3.11
Commit: 4e74f01488e03d85516b68514388c32d3c78965c
Parents: c169d49 b885e9c
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:56:22 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:56:47 2017 -0800

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/io/util/CompressedChunkReader.java| 6 --
 .../cassandra/streaming/compress/CompressedInputStream.java| 3 ++-
 3 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e74f014/CHANGES.txt
--
diff --cc CHANGES.txt
index 3c6565c,b275397..1a1a2cf
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,7 +1,16 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
  3.0.16
+  * Optimize CRC check chance probability calculations (CASSANDRA-14094)
   * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e74f014/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
--
diff --cc src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
index 8f00ce7,000..0919c29
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
+++ b/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
@@@ -1,227 -1,0 +1,229 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +
 +package org.apache.cassandra.io.util;
 +
 +import java.io.IOException;
 +import java.nio.ByteBuffer;
 +import java.util.concurrent.ThreadLocalRandom;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import com.google.common.primitives.Ints;
 +
 +import org.apache.cassandra.io.compress.BufferType;
 +import org.apache.cassandra.io.compress.CompressionMetadata;
 +import org.apache.cassandra.io.compress.CorruptBlockException;
 +import org.apache.cassandra.io.sstable.CorruptSSTableException;
 +
 +public abstract class CompressedChunkReader extends AbstractReaderFileProxy 
implements ChunkReader
 +{
 +final CompressionMetadata metadata;
 +
 +protected CompressedChunkReader(ChannelProxy channel, CompressionMetadata 
metadata)
 +{
 +super(channel, metadata.dataLength);
 +this.metadata = metadata;
 +assert Integer.bitCount(metadata.chunkLength()) == 1; //must be a 
power of two
 +}
 +
 +@VisibleForTesting
 +public double getCrcCheckChance()
 +{
 +return metadata.parameters.getCrcCheckChance();
 +}
 +
 +@Override
 +public String toString()
 +{
 +return String.format("CompressedChunkReader.%s(%s - %s, chunk length 
%d, data length %d)",
 + getClass().getSimpleName(),
 + channel.filePath(),
 +

[jira] [Updated] (CASSANDRA-13526) nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13526:
---
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   4.0
   3.11.2
   3.0.16
   Status: Resolved  (was: Ready to Commit)

Thank you so much for the patch and your patience. Committed to 3.0 as 
{{090f418831be4e4dace861fda380ee4ec27cec35}} and merged up, fixing the 3.11 
test on the way.



> nodetool cleanup on KS with no replicas should remove old data, not silently 
> complete
> -
>
> Key: CASSANDRA-13526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13526
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jeff Jirsa
>Assignee: ZhaoYang
>  Labels: usability
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> From the user list:
> https://lists.apache.org/thread.html/5d49cc6bbc6fd2e5f8b12f2308a3e24212a55afbb441af5cb8cd4167@%3Cuser.cassandra.apache.org%3E
> If you have a multi-dc cluster, but some keyspaces not replicated to a given 
> DC, you'll be unable to run cleanup on those keyspaces in that DC, because 
> [the cleanup code will see no ranges and exit 
> early|https://github.com/apache/cassandra/blob/4cfaf85/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L427-L441]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c169d491
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c169d491
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c169d491

Branch: refs/heads/cassandra-3.11
Commit: c169d491ea46abeb3ab33fbae061fd73940db6f1
Parents: f77b663 090f418
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:41:53 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:42:56 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 24 +++---
 .../org/apache/cassandra/db/CleanupTest.java| 80 
 3 files changed, 93 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c169d491/CHANGES.txt
--
diff --cc CHANGES.txt
index 8a7158d,9638886..3c6565c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,6 +1,15 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
  3.0.16
+  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c169d491/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 0a2b461,fdda562..3351736
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@@ -817,61 -651,6 +813,61 @@@ public class CompactionManager implemen
  FBUtilities.waitOnFutures(futures);
  }
  
 +public void forceUserDefinedCleanup(String dataFiles)
 +{
 +String[] filenames = dataFiles.split(",");
 +HashMap descriptors = 
Maps.newHashMap();
 +
 +for (String filename : filenames)
 +{
 +// extract keyspace and columnfamily name from filename
 +Descriptor desc = Descriptor.fromFilename(filename.trim());
 +if (Schema.instance.getCFMetaData(desc) == null)
 +{
 +logger.warn("Schema does not exist for file {}. Skipping.", 
filename);
 +continue;
 +}
 +// group by keyspace/columnfamily
 +ColumnFamilyStore cfs = 
Keyspace.open(desc.ksname).getColumnFamilyStore(desc.cfname);
 +desc = cfs.getDirectories().find(new 
File(filename.trim()).getName());
 +if (desc != null)
 +descriptors.put(cfs, desc);
 +}
 +
++if (!StorageService.instance.isJoined())
++{
++logger.error("Cleanup cannot run before a node has joined the 
ring");
++return;
++}
++
 +for (Map.Entry entry : 
descriptors.entrySet())
 +{
 +ColumnFamilyStore cfs = entry.getKey();
 +Keyspace keyspace = cfs.keyspace;
 +Collection ranges = 
StorageService.instance.getLocalRanges(keyspace.getName());
 +boolean hasIndexes = cfs.indexManager.hasIndexes();
 +SSTableReader sstable = lookupSSTable(cfs, entry.getValue());
 +
- if (ranges.isEmpty())
- {
- logger.error("Cleanup cannot run before a node has joined the 
ring");
- return;
- }
- 
 +if (sstable == null)
 +{
 +logger.warn("Will not clean {}, it is not an active sstable", 
entry.getValue());
 +}
 +else
 +{
 +CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfs, 
ranges, FBUtilities.nowInSeconds());
 +try (LifecycleTransaction txn = 
cfs.getTracker().tryModify(sstable, OperationType.CLEANUP))
 +{
 +

[3/6] cassandra git commit: Nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread jjirsa

Nodetool cleanup on KS with no replicas should remove old data, not silently 
complete

Patch by Zhao Yang; Reviewed by Jeff Jirsa for CASSANDRA-13526


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/090f4188
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/090f4188
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/090f4188

Branch: refs/heads/trunk
Commit: 090f418831be4e4dace861fda380ee4ec27cec35
Parents: 461af5b
Author: Zhao Yang 
Authored: Thu Jul 6 00:10:49 2017 +0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:40:54 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 12 ++--
 .../org/apache/cassandra/db/CleanupTest.java| 63 
 3 files changed, 70 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 54a8538..9638886 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 4483960..fdda562 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -435,12 +435,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 logger.info("Cleanup cannot run before a node has joined the 
ring");
 return AllSSTableOpStatus.ABORTED;
 }
+// if local ranges is empty, it means no data should remain
 final Collection ranges = 
StorageService.instance.getLocalRanges(keyspace.getName());
-if (ranges.isEmpty())
-{
-logger.info("Node owns no data for keyspace {}", 
keyspace.getName());
-return AllSSTableOpStatus.SUCCESSFUL;
-}
 final boolean hasIndexes = cfStore.indexManager.hasIndexes();
 
 return parallelAllSSTableOperation(cfStore, new OneSSTableOperation()
@@ -783,7 +779,10 @@ public class CompactionManager implements 
CompactionManagerMBean
 @VisibleForTesting
 public static boolean needsCleanup(SSTableReader sstable, 
Collection ownedRanges)
 {
-assert !ownedRanges.isEmpty(); // cleanup checks for this
+if (ownedRanges.isEmpty())
+{
+return true; // all data will be cleaned
+}
 
 // unwrap and sort the ranges by LHS token
 List sortedRanges = Range.normalize(ownedRanges);
@@ -842,6 +841,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 SSTableReader sstable = txn.onlyOne();
 
+// if ranges is empty and no index, entire sstable is discarded
 if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), 
sstable.last.getToken()).intersects(ranges))
 {
 txn.obsoleteOriginals();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/test/unit/org/apache/cassandra/db/CleanupTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CleanupTest.java 
b/test/unit/org/apache/cassandra/db/CleanupTest.java
index b4ffe57..99030c5 100644
--- a/test/unit/org/apache/cassandra/db/CleanupTest.java
+++ b/test/unit/org/apache/cassandra/db/CleanupTest.java
@@ -24,9 +24,11 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.util.AbstractMap;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.UUID;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
@@ -36,6 +38,8 @@ import org.junit.Test;
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.schema.KeyspaceMetadata;
 import org.apache.cassandra.cql3.Operator;
 import org.apache.cassandra.db.compaction.CompactionManager;

[1/6] cassandra git commit: Nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread jjirsa

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 461af5b9a -> 090f41883
  refs/heads/cassandra-3.11 f77b663d1 -> c169d491e
  refs/heads/trunk 0d70789fd -> e79e50b0a


Nodetool cleanup on KS with no replicas should remove old data, not silently 
complete

Patch by Zhao Yang; Reviewed by Jeff Jirsa for CASSANDRA-13526


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/090f4188
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/090f4188
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/090f4188

Branch: refs/heads/cassandra-3.0
Commit: 090f418831be4e4dace861fda380ee4ec27cec35
Parents: 461af5b
Author: Zhao Yang 
Authored: Thu Jul 6 00:10:49 2017 +0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:40:54 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 12 ++--
 .../org/apache/cassandra/db/CleanupTest.java| 63 
 3 files changed, 70 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 54a8538..9638886 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 4483960..fdda562 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -435,12 +435,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 logger.info("Cleanup cannot run before a node has joined the 
ring");
 return AllSSTableOpStatus.ABORTED;
 }
+// if local ranges is empty, it means no data should remain
 final Collection ranges = 
StorageService.instance.getLocalRanges(keyspace.getName());
-if (ranges.isEmpty())
-{
-logger.info("Node owns no data for keyspace {}", 
keyspace.getName());
-return AllSSTableOpStatus.SUCCESSFUL;
-}
 final boolean hasIndexes = cfStore.indexManager.hasIndexes();
 
 return parallelAllSSTableOperation(cfStore, new OneSSTableOperation()
@@ -783,7 +779,10 @@ public class CompactionManager implements 
CompactionManagerMBean
 @VisibleForTesting
 public static boolean needsCleanup(SSTableReader sstable, 
Collection ownedRanges)
 {
-assert !ownedRanges.isEmpty(); // cleanup checks for this
+if (ownedRanges.isEmpty())
+{
+return true; // all data will be cleaned
+}
 
 // unwrap and sort the ranges by LHS token
 List sortedRanges = Range.normalize(ownedRanges);
@@ -842,6 +841,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 SSTableReader sstable = txn.onlyOne();
 
+// if ranges is empty and no index, entire sstable is discarded
 if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), 
sstable.last.getToken()).intersects(ranges))
 {
 txn.obsoleteOriginals();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/test/unit/org/apache/cassandra/db/CleanupTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CleanupTest.java 
b/test/unit/org/apache/cassandra/db/CleanupTest.java
index b4ffe57..99030c5 100644
--- a/test/unit/org/apache/cassandra/db/CleanupTest.java
+++ b/test/unit/org/apache/cassandra/db/CleanupTest.java
@@ -24,9 +24,11 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.util.AbstractMap;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.UUID;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
@@ -36,6 +38,8 @@ import org.junit.Test;
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.config.ColumnDefinition;
+import

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e79e50b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e79e50b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e79e50b0

Branch: refs/heads/trunk
Commit: e79e50b0a784e5f6e7abebfcbfed5a49c7b2ab82
Parents: 0d70789 c169d49
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:43:13 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:45:48 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 24 +++---
 .../org/apache/cassandra/db/CleanupTest.java| 81 
 3 files changed, 94 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e79e50b0/CHANGES.txt
--
diff --cc CHANGES.txt
index cfec6d3,3c6565c..ea4ab0b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -178,7 -8,8 +178,8 @@@
   * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
  Merged from 3.0:
  3.0.16
+  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e79e50b0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e79e50b0/test/unit/org/apache/cassandra/db/CleanupTest.java
--
diff --cc test/unit/org/apache/cassandra/db/CleanupTest.java
index f576290,80e9b37..044a49e
--- a/test/unit/org/apache/cassandra/db/CleanupTest.java
+++ b/test/unit/org/apache/cassandra/db/CleanupTest.java
@@@ -35,7 -37,9 +37,9 @@@ import org.junit.Test
  
  import org.apache.cassandra.SchemaLoader;
  import org.apache.cassandra.Util;
 -import org.apache.cassandra.config.ColumnDefinition;
+ import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.schema.ColumnMetadata;
+ import org.apache.cassandra.schema.KeyspaceMetadata;
  import org.apache.cassandra.cql3.Operator;
  import org.apache.cassandra.db.compaction.CompactionManager;
  import org.apache.cassandra.db.filter.RowFilter;
@@@ -172,8 -204,55 +204,56 @@@ public class CleanupTes
  
  assertEquals(0, Util.getAll(Util.cmd(cfs).build()).size());
  }
 +
  @Test
+ public void testCleanupWithNoTokenRange() throws Exception
+ {
+ testCleanupWithNoTokenRange(false);
+ }
+ 
+ @Test
+ public void testUserDefinedCleanupWithNoTokenRange() throws Exception
+ {
+ testCleanupWithNoTokenRange(true);
+ }
+ 
+ private void testCleanupWithNoTokenRange(boolean isUserDefined) throws 
Exception
+ {
+ 
+ TokenMetadata tmd = StorageService.instance.getTokenMetadata();
+ tmd.clearUnsafe();
+ tmd.updateHostId(UUID.randomUUID(), 
InetAddress.getByName("127.0.0.1"));
+ byte[] tk1 = {2};
+ tmd.updateNormalToken(new BytesToken(tk1), 
InetAddress.getByName("127.0.0.1"));
+ 
+ 
+ Keyspace keyspace = Keyspace.open(KEYSPACE2);
+ keyspace.setMetadata(KeyspaceMetadata.create(KEYSPACE2, 
KeyspaceParams.nts("DC1", 1)));
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD2);
+ 
+ // insert data and verify we get it back w/ range query
+ fillCF(cfs, "val", LOOPS);
+ assertEquals(LOOPS, Util.getAll(Util.cmd(cfs).build()).size());
+ 
+ // remove replication on DC1
+ keyspace.setMetadata(KeyspaceMetadata.create(KEYSPACE2, 
KeyspaceParams.nts("DC1", 0)));
+ 
+ // clear token range for localhost on DC1
+ if (isUserDefined)
+ {
+ for (SSTableReader r : cfs.getLiveSSTables())
+ 
CompactionManager.instance.forceUserDefinedCleanup(r.getFilename());
+ }
+ else
+ {
+ CompactionManager.instance.performCleanup(cfs, 2);
+ }
+ assertEquals(0, Util.getAll(Util.cmd(cfs).build()).size());
+ assertTrue(cfs.getLiveSSTables().isEmpty());
+ }
+ 
+ 
+ @Test
  public void testuserDefinedCleanupWithNewToken() throws 
ExecutionException, InterruptedException, UnknownHostException
  {
  StorageService.instance.getTokenMetadata().clearUnsafe();

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-12-06 Thread jjirsa

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c169d491
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c169d491
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c169d491

Branch: refs/heads/trunk
Commit: c169d491ea46abeb3ab33fbae061fd73940db6f1
Parents: f77b663 090f418
Author: Jeff Jirsa 
Authored: Wed Dec 6 21:41:53 2017 -0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:42:56 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 24 +++---
 .../org/apache/cassandra/db/CleanupTest.java| 80 
 3 files changed, 93 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c169d491/CHANGES.txt
--
diff --cc CHANGES.txt
index 8a7158d,9638886..3c6565c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,6 +1,15 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
  3.0.16
+  * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
 - * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
 + * Fix updating base table rows with TTL not removing view entries 
(CASSANDRA-14071)
   * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c169d491/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 0a2b461,fdda562..3351736
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@@ -817,61 -651,6 +813,61 @@@ public class CompactionManager implemen
  FBUtilities.waitOnFutures(futures);
  }
  
 +public void forceUserDefinedCleanup(String dataFiles)
 +{
 +String[] filenames = dataFiles.split(",");
 +HashMap descriptors = 
Maps.newHashMap();
 +
 +for (String filename : filenames)
 +{
 +// extract keyspace and columnfamily name from filename
 +Descriptor desc = Descriptor.fromFilename(filename.trim());
 +if (Schema.instance.getCFMetaData(desc) == null)
 +{
 +logger.warn("Schema does not exist for file {}. Skipping.", 
filename);
 +continue;
 +}
 +// group by keyspace/columnfamily
 +ColumnFamilyStore cfs = 
Keyspace.open(desc.ksname).getColumnFamilyStore(desc.cfname);
 +desc = cfs.getDirectories().find(new 
File(filename.trim()).getName());
 +if (desc != null)
 +descriptors.put(cfs, desc);
 +}
 +
++if (!StorageService.instance.isJoined())
++{
++logger.error("Cleanup cannot run before a node has joined the 
ring");
++return;
++}
++
 +for (Map.Entry entry : 
descriptors.entrySet())
 +{
 +ColumnFamilyStore cfs = entry.getKey();
 +Keyspace keyspace = cfs.keyspace;
 +Collection ranges = 
StorageService.instance.getLocalRanges(keyspace.getName());
 +boolean hasIndexes = cfs.indexManager.hasIndexes();
 +SSTableReader sstable = lookupSSTable(cfs, entry.getValue());
 +
- if (ranges.isEmpty())
- {
- logger.error("Cleanup cannot run before a node has joined the 
ring");
- return;
- }
- 
 +if (sstable == null)
 +{
 +logger.warn("Will not clean {}, it is not an active sstable", 
entry.getValue());
 +}
 +else
 +{
 +CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfs, 
ranges, FBUtilities.nowInSeconds());
 +try (LifecycleTransaction txn = 
cfs.getTracker().tryModify(sstable, OperationType.CLEANUP))
 +{
 +doCleanupOne(cfs,

[2/6] cassandra git commit: Nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-12-06 Thread jjirsa

Nodetool cleanup on KS with no replicas should remove old data, not silently 
complete

Patch by Zhao Yang; Reviewed by Jeff Jirsa for CASSANDRA-13526


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/090f4188
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/090f4188
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/090f4188

Branch: refs/heads/cassandra-3.11
Commit: 090f418831be4e4dace861fda380ee4ec27cec35
Parents: 461af5b
Author: Zhao Yang 
Authored: Thu Jul 6 00:10:49 2017 +0800
Committer: Jeff Jirsa 
Committed: Wed Dec 6 21:40:54 2017 -0800

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java| 12 ++--
 .../org/apache/cassandra/db/CleanupTest.java| 63 
 3 files changed, 70 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 54a8538..9638886 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix cleanup on keyspace with no replicas (CASSANDRA-13526)
  * Fix updating base table rows with TTL not removing materialized view 
entries (CASSANDRA-14071)
  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 4483960..fdda562 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -435,12 +435,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 logger.info("Cleanup cannot run before a node has joined the 
ring");
 return AllSSTableOpStatus.ABORTED;
 }
+// if local ranges is empty, it means no data should remain
 final Collection ranges = 
StorageService.instance.getLocalRanges(keyspace.getName());
-if (ranges.isEmpty())
-{
-logger.info("Node owns no data for keyspace {}", 
keyspace.getName());
-return AllSSTableOpStatus.SUCCESSFUL;
-}
 final boolean hasIndexes = cfStore.indexManager.hasIndexes();
 
 return parallelAllSSTableOperation(cfStore, new OneSSTableOperation()
@@ -783,7 +779,10 @@ public class CompactionManager implements 
CompactionManagerMBean
 @VisibleForTesting
 public static boolean needsCleanup(SSTableReader sstable, 
Collection ownedRanges)
 {
-assert !ownedRanges.isEmpty(); // cleanup checks for this
+if (ownedRanges.isEmpty())
+{
+return true; // all data will be cleaned
+}
 
 // unwrap and sort the ranges by LHS token
 List sortedRanges = Range.normalize(ownedRanges);
@@ -842,6 +841,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 SSTableReader sstable = txn.onlyOne();
 
+// if ranges is empty and no index, entire sstable is discarded
 if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), 
sstable.last.getToken()).intersects(ranges))
 {
 txn.obsoleteOriginals();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/090f4188/test/unit/org/apache/cassandra/db/CleanupTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CleanupTest.java 
b/test/unit/org/apache/cassandra/db/CleanupTest.java
index b4ffe57..99030c5 100644
--- a/test/unit/org/apache/cassandra/db/CleanupTest.java
+++ b/test/unit/org/apache/cassandra/db/CleanupTest.java
@@ -24,9 +24,11 @@ import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
 import java.util.AbstractMap;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.UUID;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
@@ -36,6 +38,8 @@ import org.junit.Test;
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.schema.KeyspaceMetadata;
 import org.apache.cassandra.cql3.Operator;
 import

[jira] [Commented] (CASSANDRA-14054) testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is flaky: expected <2> but got <1>

2017-12-06 Thread Alex Lourie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281337#comment-16281337
 ] 

Alex Lourie commented on CASSANDRA-14054:
-

[~mkjellman] Thanks Michael, that's exactly what I was looking for! I'll need 
to go over this, but it's a really good start. Will update you after I read 
this all and have anything additional :-)

> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> -
>
> Key: CASSANDRA-14054
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14054
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Alex Lourie
>
> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> Fails about 25% of the time. It is currently our only flaky unit test on 
> trunk so it would be great to get this one fixed up so we can be confident in 
> unit test failures going forward.
> junit.framework.AssertionFailedError: Invalid value for row 0 column 0 (c of 
> type int), expected <2> but got <1>
>   at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:973)
>   at 
> org.apache.cassandra.cql3.ViewTest.testRegularColumnTimestampUpdates(ViewTest.java:380)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14054) testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is flaky: expected <2> but got <1>

2017-12-06 Thread Michael Kjellman (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281326#comment-16281326
]

Michael Kjellman edited comment on CASSANDRA-14054 at 12/7/17 4:44 AM:
---

[~alourie] hey, so sorry for the delayed reply.. i've been up to my eyeballs in
the dtest pytest work along with all the other stuff and totally let this slip.
I don't have a super great answer for you yet because I'm in the process of
getting that story together... but maybe we can make this work :)

If you take a look at my C* fork, there is a CircleCI config:
https://github.com/mkjellman/cassandra/blob/trunk_circle/.circleci/config.yml

Create a free CircleCI account (if you don't have one yet) and register your C*
fork on GitHub with CircleCI. Then, grab the above config and put it in a
branch of trunk in your personal fork (you'll need to create a .circleci folder
and put it in there.

Starting at L47 of the config you'll need to switch things to use the free user
config (i'm running under the assumption you don't have a paid CircleCI account
here).

{code}
# Set env_settings, env_vars, and workflows/build_and_run_tests based on
environment
env_settings: _settings
# <<: *default_env_settings
<<: *high_capacity_env_settings
env_vars: _vars
# <<: *default_env_vars
<<: *high_capacity_env_vars
workflows:
version: 2
# build_and_run_tests: *default_jobs
build_and_run_tests: *with_dtest_jobs
{code}

comment out the instances of high_capacity_* and comment back in the default_*
ones... and you might want to switch the workflows to only run the
"default_jobs" which for right now will just build C* and run the unit tests.

This test fails about 50% of the time on CircleCI. Potentially it's exacerbated
by running on Ubuntu? Another thing maybe worth trying is running the test via
ant on ubuntu... The docker image I put together for CircleCI is available on
DockerHub (config checked in to
https://github.com/mkjellman/cassandra-test-docker) or you can grab it as
kjellman/cassandra-test:0.1.3.

Another thing that we do is split up the unit tests across the total number of
Circle containers available... based on historical runs it actually will try to
distribute the tests that run in each container by time so you don't have a few
containers with all the slow tests dragging the entire thing down. This means
we use invoke the tests in each container via "ant testclasslist
-Dtest.classlistfile=/path/to/unit/tests/to/run"... potentially maybe another
test somewhere else doesn't clean up after itself and that causes
testRegularColumnTimestampUpdates to fail? To be clear -- the splits across
containers are on a per test method level -- not test class -- so you might
have various methods of ViewTest run across different containers at the same
time -- the results are all merged together by circle at the end to give one
consolidated report for all the unit tests. none of the other unit tests on
trunk have been flaky or failing when run via circle other than this test so
I'm not sure I totally believe it's related to order it's run in or another
test not cleaning up after itself -- also there are a lot of other asserts that
are passing before the 2nd to last assert is hit (which is the one that's
always failing -- and always failing with the same value of 1 instead of 2)...

hope all this helps get the ball rolling again... any hunches by just looking
at the code? i don't really know the MV code very well... any chance there is a
race between when the mv is completed building and available and when the
assert is hit? maybe we need some kind of force blocking flush before we assert
on those conditions? that's how we handle this in a lot of the other compaction
related tests that check sstables on disk and row count...

was (Author: mkjellman):
[~alourie] hey, so sorry for the delayed reply.. i've been up to my eyeballs in
the dtest pytest work along with all the other stuff and totally let this slip.
I don't have a super great answer for you yet because I'm in the process of
getting that story together... but maybe we can make this work :)

If you take a look at my C* fork, there is a CircleCI config:
https://github.com/mkjellman/cassandra/blob/trunk_circle/.circleci/config.yml

Starting at L47 of the config you'll need to switch things to use the free user
config (i'm running under the assumption you don't have a paid CircleCI account
here).

{code}
# Set env_settings, env_vars, and workflows/build_and_run_tests based on
environment
env_settings: _settings
# <<: *default_env_settings
<<:

[jira] [Commented] (CASSANDRA-14054) testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is flaky: expected <2> but got <1>

2017-12-06 Thread Michael Kjellman (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281326#comment-16281326
 ] 

Michael Kjellman commented on CASSANDRA-14054:
--

[~alourie] hey, so sorry for the delayed reply.. i've been up to my eyeballs in 
the dtest pytest work along with all the other stuff and totally let this slip. 
I don't have a super great answer for you yet because I'm in the process of 
getting that story together... but maybe we can make this work :)

If you take a look at my C* fork, there is a CircleCI config:
https://github.com/mkjellman/cassandra/blob/trunk_circle/.circleci/config.yml

Create a free CircleCI account (if you don't have one yet) and register your C* 
fork on GitHub with CircleCI. Then, grab the above config and put it in a 
branch of trunk in your personal fork (you'll need to create a .circleci folder 
and put it in there. 

Starting at L47 of the config you'll need to switch things to use the free user 
config (i'm running under the assumption you don't have a paid CircleCI account 
here).

{code}
# Set env_settings, env_vars, and workflows/build_and_run_tests based on 
environment
env_settings: _settings
# <<: *default_env_settings
<<: *high_capacity_env_settings
env_vars: _vars
# <<: *default_env_vars
<<: *high_capacity_env_vars
workflows:
version: 2
# build_and_run_tests: *default_jobs
build_and_run_tests: *with_dtest_jobs

comment out the instances of high_capacity_* and comment back in the default_* 
ones... and you might want to switch the workflows to only run the 
"default_jobs" which for right now will just build C* and run the unit tests.

This test fails about 50% of the time on CircleCI. Potentially it's exacerbated 
by running on Ubuntu? Another thing maybe worth trying is running the test via 
ant on ubuntu... The docker image I put together for CircleCI is available on 
DockerHub (config checked in to 
https://github.com/mkjellman/cassandra-test-docker) or you can grab it as 
kjellman/cassandra-test:0.1.3.

Another thing that we do is split up the unit tests across the total number of 
Circle containers available... based on historical runs it actually will try to 
distribute the tests that run in each container by time so you don't have a few 
containers with all the slow tests dragging the entire thing down. This means 
we use invoke the tests in each container via "ant testclasslist 
-Dtest.classlistfile=/path/to/unit/tests/to/run"... potentially maybe another 
test somewhere else doesn't clean up after itself and that causes 
testRegularColumnTimestampUpdates to fail? To be clear -- the splits across 
containers are on a per test method level -- not test class -- so you might 
have various methods of ViewTest run across different containers at the same 
time -- the results are all merged together by circle at the end to give one 
consolidated report for all the unit tests. none of the other unit tests on 
trunk have been flaky or failing when run via circle other than this test so 
I'm not sure I totally believe it's related to order it's run in or another 
test not cleaning up after itself -- also there are a lot of other asserts that 
are passing before the 2nd to last assert is hit (which is the one that's 
always failing -- and always failing with the same value of 1 instead of 2)...

hope all this helps get the ball rolling again... any hunches by just looking 
at the code? i don't really know the MV code very well... any chance there is a 
race between when the mv is completed building and available and when the 
assert is hit? maybe we need some kind of force blocking flush before we assert 
on those conditions? that's how we handle this in a lot of the other compaction 
related tests that check sstables on disk and row count...

> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> -
>
> Key: CASSANDRA-14054
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14054
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Alex Lourie
>
> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> Fails about 25% of the time. It is currently our only flaky unit test on 
> trunk so it would be great to get this one fixed up so we can be confident in 
> unit test failures going forward.
> junit.framework.AssertionFailedError: Invalid value for row 0 column 0 (c of 
> type int), expected <2> but got <1>
>   at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:973)
>   at 
> org.apache.cassandra.cql3.ViewTest.testRegularColumnTimestampUpdates(ViewTest.java:380)



--
This

[jira] [Commented] (CASSANDRA-14080) Handling 0 size hint files during start

2017-12-06 Thread Alex Lourie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281313#comment-16281313
 ] 

Alex Lourie commented on CASSANDRA-14080:
-

[~iamaleksey] Would you mind elaborating, please, which other manifestations of 
corruption you have in mind? I thought crc32 checks are supposed to cover those 
other cases.

Thanks.

> Handling 0 size hint files during start
> ---
>
> Key: CASSANDRA-14080
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14080
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints
>Reporter: Aleksandr Ivanov
>Assignee: Alex Lourie
>
> Continuation of CASSANDRA-12728 bug.
> Problem: Cassandra didn't start due to 0 size hints files
> Log form v3.0.14:
> {code:java}
> INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra 
> version: 3.0.14
> INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API 
> version: 20.1.0
> INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported 
> versions: 3.4.0 (default: 3.4.0)
> ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception 
> encountered during startup
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) 
> ~[na:1.8.0_141]
> at java.util.Iterator.forEachRemaining(Iterator.java:116) 
> ~[na:1.8.0_141]
> at 
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
>  ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) 
> ~[na:1.8.0_141]
> at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsService.(HintsService.java:88) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsService.(HintsService.java:63) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.StorageProxy.(StorageProxy.java:121) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
> at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:570)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) 
> [apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
>  [apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) 
> [apache-cassandra-3.0.14.jar:3.0.14]
> Caused by: java.io.EOFException: null
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) 
> ~[na:1.8.0_141]
> at 
> org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> ... 20 common frames omitted
> {code}
> After several 0 size hints files deletion Cassandra started successfully.
> Jeff Jirsa added a comment - Yesterday
> Aleksandr Ivanov can you open a new JIRA and link it back to this one? It's 
> possible that the original patch didn't consider 0 byte files (I don't have 
> time to go back and look at the commit, and it was long enough ago that I've 
> forgotten) - were all of your files 0 bytes?
> Not all, 8..10 hints files were with 0 size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

2017-12-06 Thread Alex Lourie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273867#comment-16273867
 ] 

Alex Lourie edited comment on CASSANDRA-13010 at 12/7/17 4:06 AM:
--

[~rustyrazorblade] I've got back to working on this ticket. I think I've 
covered all possible operations and the patch is now in a good shape.

I've tested it with compactions(including split and user-defined), repair, 
scrub, upgradesstables and cleanup operations; I also tested with multiple data 
directories. It looks ok for all of them, here are a couple of screenshots:

[^cleanup.png]
[^multiple operations.png]

I think that the patch is ready for review at github 
(https://github.com/apache/cassandra/compare/trunk...alourie:CASSANDRA-13010) 
or as a patch [^13010.patch]

Would appreciate any feedback.
Thanks.


was (Author: alourie):
[~rustyrazorblade] I've got back to working on this ticket. I think I've 
covered all possible operations and the patch is now in a good shape.

I've tested it with compactions(including split and user-defined), repair, 
scrub and cleanup operations; I also tested with multiple data directories. It 
looks ok for all of them, here are a couple of screenshots:

[^cleanup.png]
[^multiple operations.png]

I think that the patch is ready for review at github 
(https://github.com/apache/cassandra/compare/trunk...alourie:CASSANDRA-13010) 
or as a patch [^13010.patch]

Would appreciate any feedback.
Thanks.

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction, Tools
>Reporter: Jon Haddad
>Assignee: Alex Lourie
>  Labels: lhf
> Attachments: 13010.patch, cleanup.png, multiple operations.png
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14054) testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is flaky: expected <2> but got <1>

2017-12-06 Thread Alex Lourie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281281#comment-16281281
 ] 

Alex Lourie commented on CASSANDRA-14054:
-

[~mkjellman] I've finished running additional 20k cycles for this specific 
method and for the whole org.apache.cassandra.cql3.ViewTest; all tests pass 
(I'm running locally on Linux and Mac). Are there any more links, pointers or 
details that I could use to investigate it further?

At the moment I can't reproduce it.

Thanks

> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> -
>
> Key: CASSANDRA-14054
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14054
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Alex Lourie
>
> testRegularColumnTimestampUpdates - org.apache.cassandra.cql3.ViewTest is 
> flaky: expected <2> but got <1>
> Fails about 25% of the time. It is currently our only flaky unit test on 
> trunk so it would be great to get this one fixed up so we can be confident in 
> unit test failures going forward.
> junit.framework.AssertionFailedError: Invalid value for row 0 column 0 (c of 
> type int), expected <2> but got <1>
>   at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:973)
>   at 
> org.apache.cassandra.cql3.ViewTest.testRegularColumnTimestampUpdates(ViewTest.java:380)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14056) Many dtests fail with ConfigurationException: offheap_objects are not available in 3.0 when OFFHEAP_MEMTABLES="true"

2017-12-06 Thread Alex Lourie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281256#comment-16281256
 ] 

Alex Lourie commented on CASSANDRA-14056:
-

[~mkjellman] I've added a check for a specific C* version to avoid actually 
running any tests if the C* version is 3.0.x and less than 3.4; the patch is  
https://github.com/alourie/cassandra-dtest/commit/0a9661e3ed404856302ab05de4d51b2d65e9e872.
 Please have a look whether that's what you had in mind.

Thanks.

> Many dtests fail with ConfigurationException: offheap_objects are not 
> available in 3.0 when OFFHEAP_MEMTABLES="true"
> 
>
> Key: CASSANDRA-14056
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14056
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Alex Lourie
>
> Tons of dtests are running when they shouldn't as it looks like the path is 
> no longer supported.. we need to add a bunch of logic that's missing to fully 
> support running dtests with off-heap memtables enabled (via the 
> OFFHEAP_MEMTABLES="true" environment variable)
> {code}[node2 ERROR] java.lang.ExceptionInInitializerError
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:394)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:361)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:577)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:554)
>   at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:368)
>   at org.apache.cassandra.db.Keyspace.(Keyspace.java:305)
>   at org.apache.cassandra.db.Keyspace.open(Keyspace.java:129)
>   at org.apache.cassandra.db.Keyspace.open(Keyspace.java:106)
>   at 
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:887)
>   at 
> org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:354)
>   at 
> org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:110)
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:179)
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: 
> offheap_objects are not available in 3.0. They will be re-introduced in a 
> future release, see https://issues.apache.org/jira/browse/CASSANDRA-9472 for 
> details
>   at 
> org.apache.cassandra.config.DatabaseDescriptor.getMemtableAllocatorPool(DatabaseDescriptor.java:1907)
>   at org.apache.cassandra.db.Memtable.(Memtable.java:65)
>   ... 14 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14088) Forward slash in role name breaks CassandraAuthorizer

2017-12-06 Thread Jeremiah Jordan (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-14088:

Reviewer: Jeremiah Jordan

> Forward slash in role name breaks CassandraAuthorizer
> -
>
> Key: CASSANDRA-14088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14088
> Project: Cassandra
>  Issue Type: Bug
>  Components: Auth
> Environment: Git commit: 4c80eeece37d79f434078224a0504400ae10a20d 
> ({{HEAD}} of {{trunk}}).
>Reporter: Jesse Haber-Kucharsky
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The standard system authorizer 
> ({{org.apache.cassandra.auth.CassandraAuthorizer}}) stores the permissions 
> granted to each user for a given resource in {{system_auth.role_permissions}}.
> A resource like the {{my_keyspace.items}} table is stored as 
> {{"data/my_keyspace/items"}} (note the {{/}} delimiter).
> Similarly, role resources (like the {{joe}} role) are stored as 
> {{"roles/joe"}}.
> The problem is that roles can be created with {{/}} in their names, which 
> confuses the authorizer when the table is queried.
> For example,
> {code}
> $ bin/cqlsh -u cassandra -p cassandra
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 4.0-SNAPSHOT | CQL spec 3.4.5 | Native protocol v4]
> Use HELP for help.
> cassandra@cqlsh> CREATE ROLE emperor;
> cassandra@cqlsh> CREATE ROLE "ki/ng";
> cassandra@cqlsh> GRANT ALTER ON ROLE "ki/ng" TO emperor;
> cassandra@cqlsh> LIST ROLES;
>  role  | super | login | options
> ---+---+---+-
>  cassandra |  True |  True |{}
>emperor | False | False |{}
>  ki/ng | False | False |{}
> (3 rows)
> cassandra@cqlsh> SELECT * FROM system_auth.role_permissions;
>  role  | resource  | permissions
> ---+---+
>emperor |   roles/ki/ng |  {'ALTER'}
>  cassandra | roles/emperor | {'ALTER', 'AUTHORIZE', 'DROP'}
>  cassandra |   roles/ki/ng | {'ALTER', 'AUTHORIZE', 'DROP'}
> (3 rows)
> cassandra@cqlsh> LIST ALL PERMISSIONS OF emperor;
> ServerError: java.lang.IllegalArgumentException: roles/ki/ng is not a valid 
> role resource name
> {code}
> Here's the backtrace from the server process:
> {code}
> ERROR [Native-Transport-Requests-1] 2017-12-01 11:07:52,811 
> QueryMessage.java:129 - Unexpected error during query
> java.lang.IllegalArgumentException: roles/ki/ng is not a valid role resource 
> name
> at 
> org.apache.cassandra.auth.RoleResource.fromName(RoleResource.java:101) 
> ~[main/:na]
> at org.apache.cassandra.auth.Resources.fromName(Resources.java:56) 
> ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.listPermissionsForRole(CassandraAuthorizer.java:283)
>  ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.list(CassandraAuthorizer.java:263)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.list(ListPermissionsStatement.java:108)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.execute(ListPermissionsStatement.java:96)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.AuthorizationStatement.execute(AuthorizationStatement.java:48)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:207)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:238) 
> ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:223) 
> ~[main/:na]
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
>  ~[main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
>  [main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
>  [main/:na]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_151]
> at 
>

[jira] [Updated] (CASSANDRA-14088) Forward slash in role name breaks CassandraAuthorizer

2017-12-06 Thread Jeremiah Jordan (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-14088:

Status: Ready to Commit  (was: Patch Available)

> Forward slash in role name breaks CassandraAuthorizer
> -
>
> Key: CASSANDRA-14088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14088
> Project: Cassandra
>  Issue Type: Bug
>  Components: Auth
> Environment: Git commit: 4c80eeece37d79f434078224a0504400ae10a20d 
> ({{HEAD}} of {{trunk}}).
>Reporter: Jesse Haber-Kucharsky
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The standard system authorizer 
> ({{org.apache.cassandra.auth.CassandraAuthorizer}}) stores the permissions 
> granted to each user for a given resource in {{system_auth.role_permissions}}.
> A resource like the {{my_keyspace.items}} table is stored as 
> {{"data/my_keyspace/items"}} (note the {{/}} delimiter).
> Similarly, role resources (like the {{joe}} role) are stored as 
> {{"roles/joe"}}.
> The problem is that roles can be created with {{/}} in their names, which 
> confuses the authorizer when the table is queried.
> For example,
> {code}
> $ bin/cqlsh -u cassandra -p cassandra
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 4.0-SNAPSHOT | CQL spec 3.4.5 | Native protocol v4]
> Use HELP for help.
> cassandra@cqlsh> CREATE ROLE emperor;
> cassandra@cqlsh> CREATE ROLE "ki/ng";
> cassandra@cqlsh> GRANT ALTER ON ROLE "ki/ng" TO emperor;
> cassandra@cqlsh> LIST ROLES;
>  role  | super | login | options
> ---+---+---+-
>  cassandra |  True |  True |{}
>emperor | False | False |{}
>  ki/ng | False | False |{}
> (3 rows)
> cassandra@cqlsh> SELECT * FROM system_auth.role_permissions;
>  role  | resource  | permissions
> ---+---+
>emperor |   roles/ki/ng |  {'ALTER'}
>  cassandra | roles/emperor | {'ALTER', 'AUTHORIZE', 'DROP'}
>  cassandra |   roles/ki/ng | {'ALTER', 'AUTHORIZE', 'DROP'}
> (3 rows)
> cassandra@cqlsh> LIST ALL PERMISSIONS OF emperor;
> ServerError: java.lang.IllegalArgumentException: roles/ki/ng is not a valid 
> role resource name
> {code}
> Here's the backtrace from the server process:
> {code}
> ERROR [Native-Transport-Requests-1] 2017-12-01 11:07:52,811 
> QueryMessage.java:129 - Unexpected error during query
> java.lang.IllegalArgumentException: roles/ki/ng is not a valid role resource 
> name
> at 
> org.apache.cassandra.auth.RoleResource.fromName(RoleResource.java:101) 
> ~[main/:na]
> at org.apache.cassandra.auth.Resources.fromName(Resources.java:56) 
> ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.listPermissionsForRole(CassandraAuthorizer.java:283)
>  ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.list(CassandraAuthorizer.java:263)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.list(ListPermissionsStatement.java:108)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.execute(ListPermissionsStatement.java:96)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.AuthorizationStatement.execute(AuthorizationStatement.java:48)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:207)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:238) 
> ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:223) 
> ~[main/:na]
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
>  ~[main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
>  [main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
>  [main/:na]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_151]

[jira] [Commented] (CASSANDRA-14088) Forward slash in role name breaks CassandraAuthorizer

2017-12-06 Thread Jeremiah Jordan (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281250#comment-16281250
 ] 

Jeremiah Jordan commented on CASSANDRA-14088:
-

Agreed, we should just limit to only splitting the first "/".

Patch LGTM +1.

> Forward slash in role name breaks CassandraAuthorizer
> -
>
> Key: CASSANDRA-14088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14088
> Project: Cassandra
>  Issue Type: Bug
>  Components: Auth
> Environment: Git commit: 4c80eeece37d79f434078224a0504400ae10a20d 
> ({{HEAD}} of {{trunk}}).
>Reporter: Jesse Haber-Kucharsky
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The standard system authorizer 
> ({{org.apache.cassandra.auth.CassandraAuthorizer}}) stores the permissions 
> granted to each user for a given resource in {{system_auth.role_permissions}}.
> A resource like the {{my_keyspace.items}} table is stored as 
> {{"data/my_keyspace/items"}} (note the {{/}} delimiter).
> Similarly, role resources (like the {{joe}} role) are stored as 
> {{"roles/joe"}}.
> The problem is that roles can be created with {{/}} in their names, which 
> confuses the authorizer when the table is queried.
> For example,
> {code}
> $ bin/cqlsh -u cassandra -p cassandra
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 4.0-SNAPSHOT | CQL spec 3.4.5 | Native protocol v4]
> Use HELP for help.
> cassandra@cqlsh> CREATE ROLE emperor;
> cassandra@cqlsh> CREATE ROLE "ki/ng";
> cassandra@cqlsh> GRANT ALTER ON ROLE "ki/ng" TO emperor;
> cassandra@cqlsh> LIST ROLES;
>  role  | super | login | options
> ---+---+---+-
>  cassandra |  True |  True |{}
>emperor | False | False |{}
>  ki/ng | False | False |{}
> (3 rows)
> cassandra@cqlsh> SELECT * FROM system_auth.role_permissions;
>  role  | resource  | permissions
> ---+---+
>emperor |   roles/ki/ng |  {'ALTER'}
>  cassandra | roles/emperor | {'ALTER', 'AUTHORIZE', 'DROP'}
>  cassandra |   roles/ki/ng | {'ALTER', 'AUTHORIZE', 'DROP'}
> (3 rows)
> cassandra@cqlsh> LIST ALL PERMISSIONS OF emperor;
> ServerError: java.lang.IllegalArgumentException: roles/ki/ng is not a valid 
> role resource name
> {code}
> Here's the backtrace from the server process:
> {code}
> ERROR [Native-Transport-Requests-1] 2017-12-01 11:07:52,811 
> QueryMessage.java:129 - Unexpected error during query
> java.lang.IllegalArgumentException: roles/ki/ng is not a valid role resource 
> name
> at 
> org.apache.cassandra.auth.RoleResource.fromName(RoleResource.java:101) 
> ~[main/:na]
> at org.apache.cassandra.auth.Resources.fromName(Resources.java:56) 
> ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.listPermissionsForRole(CassandraAuthorizer.java:283)
>  ~[main/:na]
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.list(CassandraAuthorizer.java:263)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.list(ListPermissionsStatement.java:108)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ListPermissionsStatement.execute(ListPermissionsStatement.java:96)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.AuthorizationStatement.execute(AuthorizationStatement.java:48)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:207)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:238) 
> ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:223) 
> ~[main/:na]
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
>  ~[main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
>  [main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
>  [main/:na]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
>  [netty-all-4.1.14.Final.jar:4.1.14.Final]
> at 
>

86 matches

Mail list logo