[jira] [Updated] (CASSANDRA-14956) Paged Range Slice queries with DISTINCT can drop rows from results

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14956:

Reviewers: Marcus Eriksson

> Paged Range Slice queries with DISTINCT can drop rows from results
> --
>
> Key: CASSANDRA-14956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14956
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL/Interpreter
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
> Fix For: 2.1.x, 2.2.x
>
>
> If we have a partition where the first CQL row is fully deleted (possibly via 
> TTLs), and that partition happens to fall on the page boundary of a paged 
> range query which is using SELECT DISTINCT, the next live partition *after* 
> it is omitted from the result set. This is due to over fetching of the pages 
> and a bug in trimming those pages where overlap occurs.
> This does not affect 3.0+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743066#comment-16743066
 ] 

Alex Petrov commented on CASSANDRA-14922:
-

+1 for the latest version, huge improvement!

> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png, 
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png, 
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png, 
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example 
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1], 
> [example 
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1]) 
> because we use a small container (medium) for unit tests by default and the 
> in JVM dtests are leaking a few hundred megabytes of memory per test right 
> now. This is not a big deal because the dtest runs with the larger containers 
> continue to function fine as well as local testing as the number of in JVM 
> dtests is not yet high enough to cause a problem with more than 2GB of 
> available heap. However we should fix the memory leak so that going forwards 
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be 
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I 
> believe that we have a few potential issues that are leading to the leaks:
> 1. The 
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
>  method is not successfully cleaning up all the metrics created by the 
> {{CassandraMetricsRegistry}}
>  2. The 
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
>  method is not waiting for all the instances to finish shutting down and 
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but 
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
>  does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do 
> not leak memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14956) Paged Range Slice queries with DISTINCT can drop rows from results

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743116#comment-16743116
 ] 

Marcus Eriksson commented on CASSANDRA-14956:
-

+1, just change CASSANDRA-XYZ in the unit test comment

> Paged Range Slice queries with DISTINCT can drop rows from results
> --
>
> Key: CASSANDRA-14956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14956
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL/Interpreter
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
> Fix For: 2.1.x, 2.2.x
>
>
> If we have a partition where the first CQL row is fully deleted (possibly via 
> TTLs), and that partition happens to fall on the page boundary of a paged 
> range query which is using SELECT DISTINCT, the next live partition *after* 
> it is omitted from the result set. This is due to over fetching of the pages 
> and a bug in trimming those pages where overlap occurs.
> This does not affect 3.0+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14974) Unit test stdout capture is malfunctioning in 4.0

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740461#comment-16740461
 ] 

Benedict edited comment on CASSANDRA-14974 at 1/15/19 1:52 PM:
---

[patch|https://github.com/belliottsmith/cassandra/tree/14974], 
[CI|https://circleci.com/workflow-run/e834041e-6361-4435-a3bc-2720aa5337a9]


was (Author: benedict):
[patch|https://github.com/belliottsmith/cassandra/tree/14974], 
[CI|https://circleci.com/workflow-run/01d40d0f-52e1-429d-8cc9-22aa7fd8f07b]

> Unit test stdout capture is malfunctioning in 4.0
> -
>
> Key: CASSANDRA-14974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14974
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest, Test/unit
>Reporter: Benedict
>Assignee: Benedict
>Priority: Major
> Fix For: 4.0
>
>
> In 3.x unit tests we make sure to capture stdout to the unit test files, in 
> case tests log to stdout and not to a logger.  However, in 4.0 due to a 
> configuration parameter that is deprecated in logback, the logic is 
> short-circuited silently.
> Once fixed, this affects the cleanup of in-jvm dtests which would register an 
> infinite chain of System.out overrides (one for each node), so we make the 
> functionality robust to multiple instantiations, as well as improve its 
> startup/shutdown sequence guarantees.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14956) Paged Range Slice queries with DISTINCT can drop rows from results

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14956:

Status: Ready to Commit  (was: Patch Available)

> Paged Range Slice queries with DISTINCT can drop rows from results
> --
>
> Key: CASSANDRA-14956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14956
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL/Interpreter
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
> Fix For: 2.1.x, 2.2.x
>
>
> If we have a partition where the first CQL row is fully deleted (possibly via 
> TTLs), and that partition happens to fall on the page boundary of a paged 
> range query which is using SELECT DISTINCT, the next live partition *after* 
> it is omitted from the result set. This is due to over fetching of the pages 
> and a bug in trimming those pages where overlap occurs.
> This does not affect 3.0+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch master updated: Re-enable counter upgrade tests

[aleksey]

This is an automated email from the ASF dual-hosted git repository.

aleksey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new 6f8ced2  Re-enable counter upgrade tests
6f8ced2 is described below

commit 6f8ced2d9d04918a7ff703d4bc918bdd109c669b
Author: Aleksey Yeschenko 
AuthorDate: Tue Jan 15 10:41:10 2019 +

Re-enable counter upgrade tests
---
 upgrade_tests/cql_tests.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/upgrade_tests/cql_tests.py b/upgrade_tests/cql_tests.py
index 4350877..580addc 100644
--- a/upgrade_tests/cql_tests.py
+++ b/upgrade_tests/cql_tests.py
@@ -457,7 +457,6 @@ class TestCQL(UpgradeTester):
 res = list(cursor.execute("SELECT * FROM clicks LIMIT 4"))
 assert_length_equal(res, 4)
 
-@pytest.mark.skip("https://issues.apache.org/jira/browse/CASSANDRA-14958;)
 def test_counters(self):
 """ Validate counter support """
 cursor = self.prepare()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14974) Unit test stdout capture is malfunctioning in 4.0

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743073#comment-16743073
 ] 

Alex Petrov commented on CASSANDRA-14974:
-

+1, just need to re-run {{DatabaseDescriptorRefTest}} 

> Unit test stdout capture is malfunctioning in 4.0
> -
>
> Key: CASSANDRA-14974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14974
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest, Test/unit
>Reporter: Benedict
>Assignee: Benedict
>Priority: Major
> Fix For: 4.0
>
>
> In 3.x unit tests we make sure to capture stdout to the unit test files, in 
> case tests log to stdout and not to a logger.  However, in 4.0 due to a 
> configuration parameter that is deprecated in logback, the logic is 
> short-circuited silently.
> Once fixed, this affects the cleanup of in-jvm dtests which would register an 
> infinite chain of System.out overrides (one for each node), so we make the 
> functionality robust to multiple instantiations, as well as improve its 
> startup/shutdown sequence guarantees.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated: In JVM dtests need to clean up after instance shutdown

[benedict]

This is an automated email from the ASF dual-hosted git repository.

benedict pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 00fff3e  In JVM dtests need to clean up after instance shutdown
00fff3e is described below

commit 00fff3ee6e6c0142529de621bcaeee5790a0c235
Author: Benedict Elliott Smith 
AuthorDate: Fri Jan 11 16:09:06 2019 +

In JVM dtests need to clean up after instance shutdown

Followup - isolate executions for a given node to an executor owned by the 
instance

patch by Benedict; reviewed by Alex Petrov for CASSANDRA-14922
---
 .../async/NettyStreamingMessageSender.java |   2 +-
 .../org/apache/cassandra/utils/FBUtilities.java|  21 +++--
 .../org/apache/cassandra/distributed/Instance.java |  66 ++
 .../cassandra/distributed/InvokableInstance.java   | 100 ++---
 .../apache/cassandra/distributed/TestCluster.java  |  39 ++--
 5 files changed, 158 insertions(+), 70 deletions(-)

diff --git 
a/src/java/org/apache/cassandra/streaming/async/NettyStreamingMessageSender.java
 
b/src/java/org/apache/cassandra/streaming/async/NettyStreamingMessageSender.java
index 3fa80f5..8511a87 100644
--- 
a/src/java/org/apache/cassandra/streaming/async/NettyStreamingMessageSender.java
+++ 
b/src/java/org/apache/cassandra/streaming/async/NettyStreamingMessageSender.java
@@ -511,7 +511,7 @@ public class NettyStreamingMessageSender implements 
StreamingMessageSender
 List> futures = new 
ArrayList<>(threadToChannelMap.size());
 for (Channel channel : threadToChannelMap.values())
 futures.add(channel.close());
-FBUtilities.waitOnFutures(futures, 10 * 1000);
+FBUtilities.waitOnFutures(futures, 10, TimeUnit.SECONDS);
 threadToChannelMap.clear();
 fileTransferExecutor.shutdownNow();
 
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 8f30fce..f7aec8b 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -383,28 +383,37 @@ public class FBUtilities
 
 public static  List waitOnFutures(Iterable> futures)
 {
-return waitOnFutures(futures, -1);
+return waitOnFutures(futures, -1, null);
 }
 
 /**
- * Block for a collection of futures, with an optional timeout for each 
future.
+ * Block for a collection of futures, with optional timeout.
  *
  * @param futures
- * @param ms The number of milliseconds to wait on each future. If this 
value is less than or equal to zero,
+ * @param timeout The number of units to wait in total. If this value is 
less than or equal to zero,
  *   no tiemout value will be passed to {@link Future#get()}.
+ * @param units The units of timeout.
  */
-public static  List waitOnFutures(Iterable> futures, long ms)
+public static  List waitOnFutures(Iterable> futures, long timeout, TimeUnit units)
 {
+long endNanos = 0;
+if (timeout > 0)
+endNanos = System.nanoTime() + units.toNanos(timeout);
 List results = new ArrayList<>();
 Throwable fail = null;
 for (Future f : futures)
 {
 try
 {
-if (ms <= 0)
+if (endNanos == 0)
+{
 results.add(f.get());
+}
 else
-results.add(f.get(ms, TimeUnit.MILLISECONDS));
+{
+long waitFor = Math.max(1, endNanos - System.nanoTime());
+results.add(f.get(waitFor, TimeUnit.NANOSECONDS));
+}
 }
 catch (Throwable t)
 {
diff --git a/test/distributed/org/apache/cassandra/distributed/Instance.java 
b/test/distributed/org/apache/cassandra/distributed/Instance.java
index c68b961..0c4a260 100644
--- a/test/distributed/org/apache/cassandra/distributed/Instance.java
+++ b/test/distributed/org/apache/cassandra/distributed/Instance.java
@@ -26,6 +26,7 @@ import java.util.Collections;
 import java.util.List;
 import java.util.UUID;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.function.BiConsumer;
 
@@ -88,7 +89,7 @@ public class Instance extends InvokableInstance
 
 public Instance(InstanceConfig config, ClassLoader classLoader)
 {
-super(classLoader);
+super("node-" + config.num, classLoader);
 this.config = config;
 }
 
@@ -337,16 +338,15 @@ public class Instance extends InvokableInstance
 
 void shutdown()
 {
-runOnInstance(() -> {
+acceptsOnInstance((ExecutorService executor) -> {
 Throwable error = null;
-error = 

[cassandra] branch trunk updated: Unit test stdout capture is malfunctioning in 4.0

[benedict]

This is an automated email from the ASF dual-hosted git repository.

benedict pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new a43b651  Unit test stdout capture is malfunctioning in 4.0
a43b651 is described below

commit a43b651f8e35dd7081b8593057f118ed0c49cfd6
Author: Benedict Elliott Smith 
AuthorDate: Thu Jan 10 12:32:16 2019 -0500

Unit test stdout capture is malfunctioning in 4.0

patch by Benedict; reviewed by Alex Petrov for CASSANDRA-14974
---
 test/conf/logback-dtest.xml|   2 +-
 test/conf/logback-test.xml |   2 +-
 .../cassandra/distributed/InstanceClassLoader.java |  14 +-
 .../apache/cassandra/distributed/TestCluster.java  |   9 +
 .../apache/cassandra/LogbackStatusListener.java| 770 -
 .../config/DatabaseDescriptorRefTest.java  |  18 +-
 6 files changed, 465 insertions(+), 350 deletions(-)

diff --git a/test/conf/logback-dtest.xml b/test/conf/logback-dtest.xml
index b8019f6..7d3a8be 100644
--- a/test/conf/logback-dtest.xml
+++ b/test/conf/logback-dtest.xml
@@ -41,8 +41,8 @@
 
 
   %-5level [%thread] ${instance_id} %date{ISO8601} 
%msg%n
-  false
 
+false
   
 
   
diff --git a/test/conf/logback-test.xml b/test/conf/logback-test.xml
index 4092050..3e3349f 100644
--- a/test/conf/logback-test.xml
+++ b/test/conf/logback-test.xml
@@ -39,8 +39,8 @@
 
 
   %-5level [%thread] %date{ISO8601} %msg%n
-  false
 
+false
   
 
   
diff --git 
a/test/distributed/org/apache/cassandra/distributed/InstanceClassLoader.java 
b/test/distributed/org/apache/cassandra/distributed/InstanceClassLoader.java
index 6349d5a..c86728a 100644
--- a/test/distributed/org/apache/cassandra/distributed/InstanceClassLoader.java
+++ b/test/distributed/org/apache/cassandra/distributed/InstanceClassLoader.java
@@ -37,12 +37,12 @@ public class InstanceClassLoader extends URLClassLoader
 InstanceConfig.class,
 Message.class,
 InetAddressAndPort.class,
+InvokableInstance.SerializableBiConsumer.class,
+InvokableInstance.SerializableBiFunction.class,
 InvokableInstance.SerializableCallable.class,
-InvokableInstance.SerializableRunnable.class,
 InvokableInstance.SerializableConsumer.class,
-InvokableInstance.SerializableBiConsumer.class,
 InvokableInstance.SerializableFunction.class,
-InvokableInstance.SerializableBiFunction.class,
+InvokableInstance.SerializableRunnable.class,
 InvokableInstance.SerializableTriFunction.class,
 InvokableInstance.InstanceFunction.class
 };
@@ -98,4 +98,12 @@ public class InstanceClassLoader extends URLClassLoader
 return id -> new InstanceClassLoader(id, urls, 
commonClassNames::contains, contextClassLoader);
 }
 
+/**
+ * @return true iff this class was loaded by an InstanceClassLoader, and 
as such is used by a dtest node
+ */
+public static boolean wasLoadedByAnInstanceClassLoader(Class clazz)
+{
+return 
clazz.getClassLoader().getClass().getName().equals(InstanceClassLoader.class.getName());
+}
+
 }
diff --git a/test/distributed/org/apache/cassandra/distributed/TestCluster.java 
b/test/distributed/org/apache/cassandra/distributed/TestCluster.java
index 5056661..9f95a86 100644
--- a/test/distributed/org/apache/cassandra/distributed/TestCluster.java
+++ b/test/distributed/org/apache/cassandra/distributed/TestCluster.java
@@ -39,6 +39,10 @@ import java.util.stream.Collectors;
 
 import com.google.common.collect.Sets;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.NamedThreadFactory;
 import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.ConsistencyLevel;
 import org.apache.cassandra.db.Keyspace;
@@ -76,6 +80,11 @@ import org.apache.cassandra.utils.concurrent.SimpleCondition;
  */
 public class TestCluster implements AutoCloseable
 {
+// WARNING: we have this logger not (necessarily) for logging, but
+// to ensure we have instantiated the main classloader's LoggerFactory 
(and any LogbackStatusListener)
+// before we instantiate any for a new instance
+private static final Logger logger = 
LoggerFactory.getLogger(TestCluster.class);
+
 private final File root;
 private final List instances;
 private final Coordinator coordinator;
diff --git a/test/unit/org/apache/cassandra/LogbackStatusListener.java 
b/test/unit/org/apache/cassandra/LogbackStatusListener.java
index 756f7eb..0ec73d9 100644
--- a/test/unit/org/apache/cassandra/LogbackStatusListener.java
+++ 

[jira] [Updated] (CASSANDRA-14895) SSTable loader exception when loading 3.0/3.11 compact tables into 4.0

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14895:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Committed as 
[4c1479b5f457c3a8ed0302461ef79331cc13e798|https://github.com/apache/cassandra-dtest/commit/4c1479b5f457c3a8ed0302461ef79331cc13e798]
 to the dtests. Thanks!

Test results https://circleci.com/gh/aweisberg/cassandra/tree/14895-trunk

> SSTable loader exception when loading 3.0/3.11 compact tables into 4.0
> --
>
> Key: CASSANDRA-14895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14895
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Tools
>Reporter: Ariel Weisberg
>Assignee: Dinesh Joshi
>Priority: Blocker
>  Labels: 4.0-pre-rc-bugs
> Fix For: 4.0
>
>
> While working on the upgrade tests I added 3.0/3.11 to current tests for 
> loading old version sstables using sstable loader. [The tests for loading 
> compact sstables 
> fail.|addedhttps://github.com/apache/cassandra-dtest/blob/master/upgrade_tests/storage_engine_upgrade_test.py#L466]
> It doesn't help to alter the table to drop compact storage and then run 
> rebuild and cleanup before attempting to load into current.
> Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
> java.lang.RuntimeException: Unknown column value during deserialization
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>   at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:317)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableReader.openForBatch(SSTableReader.java:440)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.lambda$openSSTables$0(SSTableLoader.java:121)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.lambda$innerList$2(LogAwareFileLister.java:99)
>   at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>   at 
> java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
>   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>   at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>   at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:101)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
>   ... 5 more
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: 
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:96)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   ... 1 more
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>

[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 62f02802704d80496675fd3d198fac43ff24dcf7
Merge: 5ede581 ddbcff3
Author: Ariel Weisberg 
AuthorDate: Mon Jan 14 18:25:17 2019 -0500

Merge branch 'cassandra-3.0' into cassandra-3.11

 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java| 26 ++
 .../apache/cassandra/service/CassandraDaemon.java  | 17 --
 .../apache/cassandra/service/StorageService.java   | 23 +++
 .../cassandra/service/StorageServiceMBean.java |  5 +
 5 files changed, 65 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index c72e89d,bb8b54c..477afd8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.18
 +3.11.4
 + * Make stop-server.bat wait for Cassandra to terminate (CASSANDRA-14829)
 + * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
 +Merged from 3.0:
+  * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
   * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
   * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)
   * Fix cassandra-stress write hang with default options (CASSANDRA-14616)
diff --cc src/java/org/apache/cassandra/db/SystemKeyspace.java
index 973538f,541dd34..812659c
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@@ -1277,6 -1236,32 +1277,32 @@@ public final class SystemKeyspac
  executeInternal(cql, keyspace, table);
  }
  
+ /**
+  * Clears size estimates for a keyspace (used to manually clean when we 
miss a keyspace drop)
+  */
+ public static void clearSizeEstimates(String keyspace)
+ {
 -String cql = String.format("DELETE FROM %s.%s WHERE keyspace_name = 
?", NAME, SIZE_ESTIMATES);
++String cql = String.format("DELETE FROM %s.%s WHERE keyspace_name = 
?", SchemaConstants.SYSTEM_KEYSPACE_NAME, SIZE_ESTIMATES);
+ executeInternal(cql, keyspace);
+ }
+ 
+ /**
+  * @return A multimap from keyspace to table for all tables with entries 
in size estimates
+  */
+ 
+ public static synchronized SetMultimap 
getTablesWithSizeEstimates()
+ {
+ SetMultimap keyspaceTableMap = HashMultimap.create();
 -String cql = String.format("SELECT keyspace_name, table_name FROM 
%s.%s", NAME, SIZE_ESTIMATES);
++String cql = String.format("SELECT keyspace_name, table_name FROM 
%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SIZE_ESTIMATES);
+ UntypedResultSet rs = executeInternal(cql);
+ for (UntypedResultSet.Row row : rs)
+ {
+ keyspaceTableMap.put(row.getString("keyspace_name"), 
row.getString("table_name"));
+ }
+ 
+ return keyspaceTableMap;
+ }
+ 
  public static synchronized void updateAvailableRanges(String keyspace, 
Collection> completedRanges)
  {
  String cql = "UPDATE system.%s SET ranges = ranges + ? WHERE 
keyspace_name = ?";
diff --cc src/java/org/apache/cassandra/service/CassandraDaemon.java
index fbafd35,9a3a414..b593190
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@@ -337,10 -308,56 +337,19 @@@ public class CassandraDaemo
  // migrate any legacy (pre-3.0) batch entries from system.batchlog to 
system.batches (new table format)
  LegacyBatchlogMigrator.migrate();
  
 -// enable auto compaction
 -for (Keyspace keyspace : Keyspace.all())
 -{
 -for (ColumnFamilyStore cfs : keyspace.getColumnFamilyStores())
 -{
 -for (final ColumnFamilyStore store : cfs.concatWithIndexes())
 -{
 -if 
(store.getCompactionStrategyManager().shouldBeEnabled())
 -store.enableAutoCompaction();
 -}
 -}
 -}
 -
 -Runnable viewRebuild = new Runnable()
 -{
 -@Override
 -public void run()
 -{
 -for (Keyspace keyspace : Keyspace.all())
 -{
 -keyspace.viewManager.buildAllViews();
 -}
 -logger.debug("Completed submission of build tasks for any 
materialized views defined at startup");
 -}
 -};
 -
 -ScheduledExecutors.optionalTasks.schedule(viewRebuild, 
StorageService.RING_DELAY, TimeUnit.MILLISECONDS);
 -
  SystemKeyspace.finishStartup();
  
+ // Clean up system.size_estimates entries left lying around from 
missed keyspace drops (CASSANDRA-14905)
+ 

[cassandra] branch cassandra-3.0 updated: If SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new ddbcff3  If SizeEstimatesRecorder misses a 'onDropTable' notification, 
the size_estimates table will never be cleared for that table.
ddbcff3 is described below

commit ddbcff3363c5ad13bd8975e80b3f28ae8149a459
Author: Joel Knighton 
AuthorDate: Tue Sep 12 17:48:07 2017 -0500

If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table.

Patch by Joel Knighton; Reviewed by Ariel Weisberg for CASSANDRA-14905

Co-authored-by: Joel Knighton 
Co-authored-by: Aleksandr Sorokoumov 
---
 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java| 26 ++
 .../apache/cassandra/service/CassandraDaemon.java  | 16 +++--
 .../apache/cassandra/service/StorageService.java   | 23 +++
 .../cassandra/service/StorageServiceMBean.java |  5 +
 5 files changed, 64 insertions(+), 7 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 063e8fb..bb8b54c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.18
+ * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
  * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
  * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)
  * Fix cassandra-stress write hang with default options (CASSANDRA-14616)
diff --git a/src/java/org/apache/cassandra/db/SystemKeyspace.java 
b/src/java/org/apache/cassandra/db/SystemKeyspace.java
index 5f5041c..541dd34 100644
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@ -1236,6 +1236,32 @@ public final class SystemKeyspace
 executeInternal(cql, keyspace, table);
 }
 
+/**
+ * Clears size estimates for a keyspace (used to manually clean when we 
miss a keyspace drop)
+ */
+public static void clearSizeEstimates(String keyspace)
+{
+String cql = String.format("DELETE FROM %s.%s WHERE keyspace_name = 
?", NAME, SIZE_ESTIMATES);
+executeInternal(cql, keyspace);
+}
+
+/**
+ * @return A multimap from keyspace to table for all tables with entries 
in size estimates
+ */
+
+public static synchronized SetMultimap 
getTablesWithSizeEstimates()
+{
+SetMultimap keyspaceTableMap = HashMultimap.create();
+String cql = String.format("SELECT keyspace_name, table_name FROM 
%s.%s", NAME, SIZE_ESTIMATES);
+UntypedResultSet rs = executeInternal(cql);
+for (UntypedResultSet.Row row : rs)
+{
+keyspaceTableMap.put(row.getString("keyspace_name"), 
row.getString("table_name"));
+}
+
+return keyspaceTableMap;
+}
+
 public static synchronized void updateAvailableRanges(String keyspace, 
Collection> completedRanges)
 {
 String cql = "UPDATE system.%s SET ranges = ranges + ? WHERE 
keyspace_name = ?";
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java 
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 6869d2c..9a3a414 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -336,9 +336,17 @@ public class CassandraDaemon
 
 ScheduledExecutors.optionalTasks.schedule(viewRebuild, 
StorageService.RING_DELAY, TimeUnit.MILLISECONDS);
 
-
 SystemKeyspace.finishStartup();
 
+// Clean up system.size_estimates entries left lying around from 
missed keyspace drops (CASSANDRA-14905)
+StorageService.instance.cleanupSizeEstimates();
+
+// schedule periodic dumps of table size estimates into 
SystemKeyspace.SIZE_ESTIMATES_CF
+// set cassandra.size_recorder_interval to 0 to disable
+int sizeRecorderInterval = 
Integer.getInteger("cassandra.size_recorder_interval", 5 * 60);
+if (sizeRecorderInterval > 0)
+
ScheduledExecutors.optionalTasks.scheduleWithFixedDelay(SizeEstimatesRecorder.instance,
 30, sizeRecorderInterval, TimeUnit.SECONDS);
+
 // start server internals
 StorageService.instance.registerDaemon(this);
 try
@@ -380,12 +388,6 @@ public class CassandraDaemon
 // due to scheduling errors or race conditions
 
ScheduledExecutors.optionalTasks.scheduleWithFixedDelay(ColumnFamilyStore.getBackgroundCompactionTaskSubmitter(),
 5, 1, TimeUnit.MINUTES);
 
-// schedule periodic dumps of table size estimates into 
SystemKeyspace.SIZE_ESTIMATES_CF
-// set cassandra.size_recorder_interval to 0 to disable
-   

[cassandra] branch cassandra-3.11 updated (5ede581 -> 62f0280)

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 5ede581  Merge branch 'cassandra-3.0' into cassandra-3.11
 new ddbcff3  If SizeEstimatesRecorder misses a 'onDropTable' notification, 
the size_estimates table will never be cleared for that table.
 new 62f0280  Merge branch 'cassandra-3.0' into cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java| 26 ++
 .../apache/cassandra/service/CassandraDaemon.java  | 17 --
 .../apache/cassandra/service/StorageService.java   | 23 +++
 .../cassandra/service/StorageServiceMBean.java |  5 +
 5 files changed, 65 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14983) Local reads potentially blocking remote reads

[Marcus Olsson (JIRA)]

Marcus Olsson created CASSANDRA-14983:
-

 Summary: Local reads potentially blocking remote reads
 Key: CASSANDRA-14983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14983
 Project: Cassandra
  Issue Type: Bug
  Components: Consistency/Coordination
Reporter: Marcus Olsson
 Attachments: local_read_trace.log

Since CASSANDRA-4718 there is a fast path allowing local requests to continue 
to [work in the same 
thread|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L157]
 rather than being sent over to the read stage.

Based on the comment
{code:java}
// We delay the local (potentially blocking) read till the end to avoid 
stalling remote requests.
{code}
it seems like this should be performed last in the chain to avoid blocking 
remote requests but that does not seem to be the case when the local request is 
a data request. The digest request(s) are sent after the data requests are sent 
(and now the transient replica requests as well). When the fast path is used 
for local data/transient data requests this will block the next type of request 
from being sent away until the local read is finished and add additional 
latency to the request.

In addition to this it seems like local requests are *always* data requests 
(might not be a problem), but the log message can say either ["digest" or 
"data"|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L156]
 as the type of request.

I have tried to run performance measurements to see the impact of this in 3.0 
(by moving local requests to the end of ARE#executeAsync()) but I haven't seen 
any big difference yet. I'll continue to run some more tests to see if I can 
find a use case affected by this.

Attaching a trace (3.0) where this happens. Reproduction:
 # Create a three node CCM cluster
 # Provision data with stress (rf=3)
 # In parallel:
 ## Start stress read run
 ## Run multiple manual read queries in cqlsh with tracing on and local_quorum 
(as this does not always happen)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14895) SSTable loader exception when loading 3.0/3.11 compact tables into 4.0

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743196#comment-16743196
 ] 

Ariel Weisberg commented on CASSANDRA-14895:


All good, going to commit this today. I just lost track of where I had run the 
tests last time so I am running them again. Took a while because it was both 
upgrade and dtests.

> SSTable loader exception when loading 3.0/3.11 compact tables into 4.0
> --
>
> Key: CASSANDRA-14895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14895
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Tools
>Reporter: Ariel Weisberg
>Assignee: Dinesh Joshi
>Priority: Blocker
>  Labels: 4.0-pre-rc-bugs
> Fix For: 4.0
>
>
> While working on the upgrade tests I added 3.0/3.11 to current tests for 
> loading old version sstables using sstable loader. [The tests for loading 
> compact sstables 
> fail.|addedhttps://github.com/apache/cassandra-dtest/blob/master/upgrade_tests/storage_engine_upgrade_test.py#L466]
> It doesn't help to alter the table to drop compact storage and then run 
> rebuild and cleanup before attempting to load into current.
> Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
> java.lang.RuntimeException: Unknown column value during deserialization
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>   at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:317)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableReader.openForBatch(SSTableReader.java:440)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.lambda$openSSTables$0(SSTableLoader.java:121)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.lambda$innerList$2(LogAwareFileLister.java:99)
>   at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>   at 
> java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
>   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>   at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>   at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:101)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
>   ... 5 more
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: 
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:96)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   ... 1 more
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>   at 
> 

[cassandra-dtest] branch master updated: SSTable loader exception when loading 3.0/3.11 compact tables into 4.0

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new 4c1479b  SSTable loader exception when loading 3.0/3.11 compact tables 
into 4.0
4c1479b is described below

commit 4c1479b5f457c3a8ed0302461ef79331cc13e798
Author: Dinesh A. Joshi 
AuthorDate: Mon Jan 14 17:39:16 2019 -0500

SSTable loader exception when loading 3.0/3.11 compact tables into 4.0

Patch by Dinesh Joshi; Reviewed by Ariel Weisberg for CASSANDRA-14895
---
 dtest.py | 14 --
 sstable_generation_loading_test.py   | 16 ++--
 upgrade_tests/storage_engine_upgrade_test.py |  2 --
 3 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/dtest.py b/dtest.py
index 9027d75..ec79771 100644
--- a/dtest.py
+++ b/dtest.py
@@ -270,6 +270,7 @@ class Tester:
 except TimeoutError:
 pytest.fail("Log message was not seen within 
timeout:\n{0}".format(msg))
 
+
 def get_eager_protocol_version(cassandra_version):
 """
 Returns the highest protocol version accepted
@@ -288,7 +289,8 @@ def get_eager_protocol_version(cassandra_version):
 
 # We default to UTF8Type because it's simpler to use in tests
 def create_cf(session, name, key_type="varchar", speculative_retry=None, 
read_repair=None, compression=None,
-  gc_grace=None, columns=None, validation="UTF8Type", 
compact_storage=False, compaction_strategy='SizeTieredCompactionStrategy'):
+  gc_grace=None, columns=None, validation="UTF8Type", 
compact_storage=False, compaction_strategy='SizeTieredCompactionStrategy',
+  primary_key=None, clustering=None):
 
 compaction_fragment = "compaction = {'class': '%s', 'enabled': 'true'}"
 if compaction_strategy == '':
@@ -304,11 +306,17 @@ def create_cf(session, name, key_type="varchar", 
speculative_retry=None, read_re
 if additional_columns == "":
 query = 'CREATE COLUMNFAMILY %s (key %s, c varchar, v varchar, PRIMARY 
KEY(key, c)) WITH comment=\'test cf\'' % (name, key_type)
 else:
-query = 'CREATE COLUMNFAMILY %s (key %s PRIMARY KEY%s) WITH 
comment=\'test cf\'' % (name, key_type, additional_columns)
+if primary_key:
+query = 'CREATE COLUMNFAMILY %s (key %s%s, PRIMARY KEY(%s)) WITH 
comment=\'test cf\'' % (name, key_type, additional_columns, primary_key)
+else:
+query = 'CREATE COLUMNFAMILY %s (key %s PRIMARY KEY%s) WITH 
comment=\'test cf\'' % (name, key_type, additional_columns)
 
 if compaction_fragment is not None:
 query = '%s AND %s' % (query, compaction_fragment)
 
+if clustering:
+query = '%s AND CLUSTERING ORDER BY (%s)' % (query, clustering)
+
 if compression is not None:
 query = '%s AND compression = { \'sstable_compression\': 
\'%sCompressor\' }' % (query, compression)
 else:
@@ -333,6 +341,7 @@ def create_cf(session, name, key_type="varchar", 
speculative_retry=None, read_re
 #Going to ignore OperationTimedOut from create CF, so need to validate it 
was indeed created
 session.execute('SELECT * FROM %s LIMIT 1' % name);
 
+
 def create_cf_simple(session, name, query):
 try:
 retry_till_success(session.execute, query=query, timeout=120, 
bypassed_exception=cassandra.OperationTimedOut)
@@ -342,6 +351,7 @@ def create_cf_simple(session, name, query):
 #Going to ignore OperationTimedOut from create CF, so need to validate it 
was indeed created
 session.execute('SELECT * FROM %s LIMIT 1' % name)
 
+
 def create_ks(session, name, rf):
 query = 'CREATE KEYSPACE %s WITH replication={%s}'
 if isinstance(rf, int):
diff --git a/sstable_generation_loading_test.py 
b/sstable_generation_loading_test.py
index 901011d..119f078 100644
--- a/sstable_generation_loading_test.py
+++ b/sstable_generation_loading_test.py
@@ -43,6 +43,14 @@ class TestBaseSStableLoader(Tester):
 if self.__class__.__name__ != 'TestBasedSSTableLoader' and 
self.upgrade_from is None:
 pytest.skip("Don't need to run base class test, only derived 
classes")
 
+def create_schema_40(self, session, ks, compression):
+create_ks(session, ks, rf=2)
+create_cf(session, "standard1", compression=compression, 
compact_storage=self.compact())
+create_cf(session, "counter1", key_type='text', 
compression=compression, columns={'column1': 'text',
+   
 'v': 'counter static',
+   
 'value': 'counter'},
+  primary_key="key, column1", clustering='column1 ASC', 
compact_storage=self.compact())
+
 def test_sstableloader_compression_none_to_none(self):
 self.skip_base_class_test()
 

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 028b6772231e18e2357dfff1fa04477efa377730
Merge: 00fff3e 62f0280
Author: Ariel Weisberg 
AuthorDate: Tue Jan 15 11:03:53 2019 -0500

Merge branch 'cassandra-3.11' into trunk

 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java| 26 +-
 .../apache/cassandra/service/CassandraDaemon.java  | 16 -
 .../apache/cassandra/service/StorageService.java   | 23 +++
 .../cassandra/service/StorageServiceMBean.java |  5 +
 5 files changed, 64 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index ba44dcf,477afd8..5af5b64
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -343,8 -2,11 +343,9 @@@
   * Make stop-server.bat wait for Cassandra to terminate (CASSANDRA-14829)
   * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
  Merged from 3.0:
+  * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
 - * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
   * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)
   * Fix cassandra-stress write hang with default options (CASSANDRA-14616)
 - * Differentiate between slices and RTs when decoding legacy bounds 
(CASSANDRA-14919)
   * Netty epoll IOExceptions caused by unclean client disconnects being logged 
at INFO (CASSANDRA-14909)
   * Unfiltered.isEmpty conflicts with Row extends AbstractCollection.isEmpty 
(CASSANDRA-14588)
   * RangeTombstoneList doesn't properly clean up mergeable or superseded rts 
in some cases (CASSANDRA-14894)
diff --cc src/java/org/apache/cassandra/db/SystemKeyspace.java
index 1b3b2a6,812659c..ddf6475
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@@ -777,7 -802,7 +777,7 @@@ public final class SystemKeyspac
  
  /**
   * This method is used to update the System Keyspace with the new tokens 
for this node
--*/
++ */
  public static synchronized void updateTokens(Collection tokens)
  {
  assert !tokens.isEmpty() : "removeEndpoint should be used instead";
@@@ -1275,49 -1277,57 +1275,73 @@@
  executeInternal(cql, keyspace, table);
  }
  
+ /**
+  * Clears size estimates for a keyspace (used to manually clean when we 
miss a keyspace drop)
+  */
+ public static void clearSizeEstimates(String keyspace)
+ {
+ String cql = String.format("DELETE FROM %s.%s WHERE keyspace_name = 
?", SchemaConstants.SYSTEM_KEYSPACE_NAME, SIZE_ESTIMATES);
+ executeInternal(cql, keyspace);
+ }
+ 
+ /**
+  * @return A multimap from keyspace to table for all tables with entries 
in size estimates
+  */
 -
+ public static synchronized SetMultimap 
getTablesWithSizeEstimates()
+ {
+ SetMultimap keyspaceTableMap = HashMultimap.create();
+ String cql = String.format("SELECT keyspace_name, table_name FROM 
%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SIZE_ESTIMATES);
+ UntypedResultSet rs = executeInternal(cql);
+ for (UntypedResultSet.Row row : rs)
+ {
+ keyspaceTableMap.put(row.getString("keyspace_name"), 
row.getString("table_name"));
+ }
 -
+ return keyspaceTableMap;
+ }
+ 
 -public static synchronized void updateAvailableRanges(String keyspace, 
Collection> completedRanges)
 +public static synchronized void updateAvailableRanges(String keyspace, 
Collection> completedFullRanges, Collection> 
completedTransientRanges)
  {
 -String cql = "UPDATE system.%s SET ranges = ranges + ? WHERE 
keyspace_name = ?";
 -Set rangesToUpdate = new 
HashSet<>(completedRanges.size());
 -for (Range range : completedRanges)
 -{
 -rangesToUpdate.add(rangeToBytes(range));
 -}
 -executeInternal(String.format(cql, AVAILABLE_RANGES), rangesToUpdate, 
keyspace);
 +String cql = "UPDATE system.%s SET full_ranges = full_ranges + ?, 
transient_ranges = transient_ranges + ? WHERE keyspace_name = ?";
 +executeInternal(format(cql, AVAILABLE_RANGES_V2),
 +
completedFullRanges.stream().map(SystemKeyspace::rangeToBytes).collect(Collectors.toSet()),
 +
completedTransientRanges.stream().map(SystemKeyspace::rangeToBytes).collect(Collectors.toSet()),
 +keyspace);
  }
  
 -public static synchronized Set> getAvailableRanges(String 
keyspace, IPartitioner partitioner)
 +/**
 + * List of the streamed ranges, where transientness is encoded based on 
the source, where range was streamed from.
 + */
 +public static synchronized 

[cassandra] branch trunk updated (00fff3e -> 028b677)

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 00fff3e  In JVM dtests need to clean up after instance shutdown
 add ddbcff3  If SizeEstimatesRecorder misses a 'onDropTable' notification, 
the size_estimates table will never be cleared for that table.
 add 62f0280  Merge branch 'cassandra-3.0' into cassandra-3.11
 new 028b677  Merge branch 'cassandra-3.11' into trunk

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java| 26 +-
 .../apache/cassandra/service/CassandraDaemon.java  | 16 -
 .../apache/cassandra/service/StorageService.java   | 23 +++
 .../cassandra/service/StorageServiceMBean.java |  5 +
 5 files changed, 64 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14526) dtest to validate Cassandra state post failed/successful bootstrap

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743272#comment-16743272
 ] 

Jaydeepkumar Chovatia commented on CASSANDRA-14526:
---

Thanks [~jay.zhuang]!

> dtest to validate Cassandra state post failed/successful bootstrap
> --
>
> Key: CASSANDRA-14526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14526
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Test/dtest
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
>  Labels: dtest
>
> Please find dtest here:
> || dtest ||
> | [patch 
> |https://github.com/apache/cassandra-dtest/compare/master...jaydeepkumar1984:14526-trunk]|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14985) CircleCI docker image should bake in more dependencies

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14985:
--

 Summary:  CircleCI docker image  should bake in more dependencies
 Key: CASSANDRA-14985
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14985
 Project: Cassandra
  Issue Type: Test
  Components: Test/dtest, Test/unit
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
 Fix For: 4.0


It's pretty frequent especially in the upgrade tests for either maven or github 
to fail. I think they are detecting the thundering herd of containers all 
pulling stuff at the same time and opting out.

We can reduce this especially for maven dependencies since most are hardly ever 
changing by downloading everything we can in advance and baking it into the 
image.

When building the docker image Initialize the local maven repository by running 
ant maven-ant-tasks-retrieve-build for 2.1-trunk and do the same with CCM and 
the Apache repository.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch master updated: Check nativetransport while bootstrap not complete

[jzhuang]

This is an automated email from the ASF dual-hosted git repository.

jzhuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new e6f58cb  Check nativetransport while bootstrap not complete
e6f58cb is described below

commit e6f58cb33f7a09f273c5990d5d21c7b529ba80bf
Author: jaydeepkumar1984 
AuthorDate: Sun Jun 17 22:01:46 2018 -0700

Check nativetransport while bootstrap not complete

patch by Jaydeepkumar Chovatia; reviewed by Jay Zhuang for CASSANDRA-14526
---
 bootstrap_test.py | 103 +-
 byteman/pre4.0/stream_failure.btm |   2 +-
 secondary_indexes_test.py |   5 +-
 tools/assertions.py   |   6 ++-
 4 files changed, 98 insertions(+), 18 deletions(-)

diff --git a/bootstrap_test.py b/bootstrap_test.py
index 6e7682f..e33749e 100644
--- a/bootstrap_test.py
+++ b/bootstrap_test.py
@@ -10,7 +10,8 @@ import signal
 
 from cassandra import ConsistencyLevel
 from cassandra.concurrent import execute_concurrent_with_args
-from ccmlib.node import NodeError
+from ccmlib.node import NodeError, TimeoutError, ToolError
+from ccmlib.node import TimeoutError
 
 import pytest
 
@@ -20,12 +21,14 @@ from tools.assertions import (assert_almost_equal, 
assert_bootstrap_state, asser
 from tools.data import query_c1c2
 from tools.intervention import InterruptBootstrap, KillOnBootstrap
 from tools.misc import new_node
-from tools.misc import generate_ssl_stores, retry_till_success
+from tools.misc import generate_ssl_stores
 
 since = pytest.mark.since
 logger = logging.getLogger(__name__)
 
 class TestBootstrap(Tester):
+byteman_submit_path_pre_4_0 = './byteman/pre4.0/stream_failure.btm'
+byteman_submit_path_4_0 = './byteman/4.0/stream_failure.btm'
 
 @pytest.fixture(autouse=True)
 def fixture_add_additional_log_patterns(self, fixture_dtest_setup):
@@ -308,26 +311,22 @@ class TestBootstrap(Tester):
 cluster.start(wait_other_notice=True)
 # kill stream to node3 in the middle of streaming to let it fail
 if cluster.version() < '4.0':
-node1.byteman_submit(['./byteman/pre4.0/stream_failure.btm'])
+node1.byteman_submit([self.byteman_submit_path_pre_4_0])
 else:
-node1.byteman_submit(['./byteman/4.0/stream_failure.btm'])
+node1.byteman_submit([self.byteman_submit_path_4_0])
 node1.stress(['write', 'n=1K', 'no-warmup', 'cl=TWO', '-schema', 
'replication(factor=2)', '-rate', 'threads=50'])
 cluster.flush()
 
 # start bootstrapping node3 and wait for streaming
 node3 = new_node(cluster)
-node3.start(wait_other_notice=False, wait_for_binary_proto=True)
+node3.start(wait_other_notice=False)
 
-# wait for node3 ready to query
-node3.watch_log_for("Starting listening for CQL clients")
-mark = node3.mark_log()
-# check if node3 is still in bootstrap mode
-retry_till_success(assert_bootstrap_state, tester=self, node=node3, 
expected_bootstrap_state='IN_PROGRESS', timeout=120)
+# let streaming fail as we expect
+node3.watch_log_for('Some data streaming failed')
 
-# bring back node1 and invoke nodetool bootstrap to resume 
bootstrapping
+# bring back node3 and invoke nodetool bootstrap to resume 
bootstrapping
 node3.nodetool('bootstrap resume')
-
-node3.watch_log_for("Resume complete", from_mark=mark)
+node3.wait_for_binary_interface()
 assert_bootstrap_state(self, node3, 'COMPLETED')
 
 # cleanup to guarantee each node will only have sstables of its ranges
@@ -706,3 +705,81 @@ class TestBootstrap(Tester):
 logger.debug("Deleting {}".format(data_dir))
 shutil.rmtree(data_dir)
 shutil.rmtree(commitlog_dir)
+
+@since('2.2')
+def test_bootstrap_binary_disabled(self):
+"""
+Test binary while bootstrapping and streaming fails
+@jira_ticket CASSANDRA-14526, CASSANDRA-14525
+"""
+config = {'authenticator': 
'org.apache.cassandra.auth.PasswordAuthenticator',
+  'authorizer': 
'org.apache.cassandra.auth.CassandraAuthorizer',
+  'role_manager': 
'org.apache.cassandra.auth.CassandraRoleManager',
+  'permissions_validity_in_ms': 0,
+  'roles_validity_in_ms': 0}
+
+cluster = self.cluster
+cluster.populate(1)
+
+node1 = cluster.nodes['node1']
+# set up byteman
+node1.byteman_port = '8100'
+node1.import_config_files()
+
+cluster.start(wait_other_notice=True)
+# kill stream to node2 in the middle of streaming to let it fail
+if cluster.version() < '4.0':
+node1.byteman_submit([self.byteman_submit_path_pre_4_0])
+else:
+

[jira] [Commented] (CASSANDRA-14895) SSTable loader exception when loading 3.0/3.11 compact tables into 4.0

[Dinesh Joshi (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743267#comment-16743267
 ] 

Dinesh Joshi commented on CASSANDRA-14895:
--

Thanks, [~aweisberg]

> SSTable loader exception when loading 3.0/3.11 compact tables into 4.0
> --
>
> Key: CASSANDRA-14895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14895
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Tools
>Reporter: Ariel Weisberg
>Assignee: Dinesh Joshi
>Priority: Blocker
>  Labels: 4.0-pre-rc-bugs
> Fix For: 4.0
>
>
> While working on the upgrade tests I added 3.0/3.11 to current tests for 
> loading old version sstables using sstable loader. [The tests for loading 
> compact sstables 
> fail.|addedhttps://github.com/apache/cassandra-dtest/blob/master/upgrade_tests/storage_engine_upgrade_test.py#L466]
> It doesn't help to alter the table to drop compact storage and then run 
> rebuild and cleanup before attempting to load into current.
> Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
> java.lang.RuntimeException: Unknown column value during deserialization
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>   at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:317)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableReader.openForBatch(SSTableReader.java:440)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.lambda$openSSTables$0(SSTableLoader.java:121)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.lambda$innerList$2(LogAwareFileLister.java:99)
>   at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>   at 
> java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
>   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>   at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>   at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:101)
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
>   ... 5 more
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: 
> java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:96)
>   at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: java.lang.RuntimeException: Failed to list files in 
> /var/folders/vx/2fcrbbx12g9bppxk7h41ww70gn/T/dtest-4_4vb5jj/test/node1/data1_copy/ks/counter1-f4dc7fc0e91011e892e9c3e97b26557e
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
>   at 
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:166)
>   at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
>   ... 1 more
> Caused by: java.lang.RuntimeException: Unknown column value during 
> deserialization
>   at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:317)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableReader.openForBatch(SSTableReader.java:440)
>   at 
> 

[jira] [Resolved] (CASSANDRA-14815) SEPExecutor does not fully shut down

[Benedict (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benedict resolved CASSANDRA-14815.
--
Resolution: Fixed

The backport work for this is being tracked/handled in CASSANDRA-14931.

> SEPExecutor does not fully shut down
> 
>
> Key: CASSANDRA-14815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14815
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Startup and Shutdown
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Minor
> Fix For: 4.0
>
>
> When trying to shut down an SEP Executor, a parked worked will still be 
> parked on:
> {code}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:88)
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-2.2 updated (07558a3 -> 900292b)

[benedict]

This is an automated email from the ASF dual-hosted git repository.

benedict pushed a change to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


 discard 07558a3  circleci

This update removed existing revisions from the reference, leaving the
reference pointing at a previous point in the repository history.

 * -- * -- N   refs/heads/cassandra-2.2 (900292b)
\
 O -- O -- O   (07558a3)

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .circleci/config.yml | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-2.2 updated: circleci

[benedict]

This is an automated email from the ASF dual-hosted git repository.

benedict pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
 new 07558a3  circleci
07558a3 is described below

commit 07558a383065a2fac85df83993589bafd6cf1b9f
Author: Benedict Elliott Smith 
AuthorDate: Tue Jan 15 19:20:53 2019 +

circleci
---
 .circleci/config.yml | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 78dae4c..b0ea655 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -64,16 +64,16 @@ upgrade_job_only: _job_only
   - build
 # Set env_settings, env_vars, and workflows/build_and_run_tests based on 
environment
 env_settings: _settings
-<<: *default_env_settings
-#<<: *high_capacity_env_settings
+#<<: *default_env_settings
+<<: *high_capacity_env_settings
 env_vars: _vars
-<<: *resource_constrained_env_vars
-#<<: *high_capacity_env_vars
+#<<: *resource_constrained_env_vars
+<<: *high_capacity_env_vars
 workflows:
 version: 2
-build_and_run_tests: *default_jobs
+#build_and_run_tests: *default_jobs
 #build_and_run_tests: *with_dtest_jobs_only
-#build_and_run_tests: *with_dtest_jobs
+build_and_run_tests: *with_dtest_jobs
 #build_and_run_tests: *upgrade_job_only
 docker_image: _image spod/cassandra-testing-ubuntu18-java11
 version: 2


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14975) cqlsh dtests are not running in CircleCI

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14975:
---
Fix Version/s: 4.0.x
   3.11.x
   3.0.x
   Status: Patch Available  (was: Open)

I got it done and I didn't have to skip any tests. I made a few bits of cqlsh 
python 3 compatible. If we want to move tests around it's a bit of an 
orthogonal concern to getting the tests we have running. You can move the 
working tests around wherever you want.

[dtest 
code|https://github.com/apache/cassandra-dtest/compare/master...aweisberg:14975?expand=1]
[trunk|https://github.com/apache/cassandra/compare/trunk...aweisberg:14975-trunk?expand=1]
[tests|https://circleci.com/gh/aweisberg/cassandra/tree/14975-trunk]

The trunk patch cherry-picks cleanly to 2.2, 3.0, and 3.11. I don't really want 
to change 2.2 so I am thinking of starting at 3.0. I'll test 3.0, and 3.11 
shortly.

> cqlsh dtests are not running in CircleCI
> 
>
> Key: CASSANDRA-14975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14975
> Project: Cassandra
>  Issue Type: Test
>  Components: Test/dtest
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Right now the dtests skip these tests because most don't pass. Originally 
> they weren't being collected because the test files end in_tests.py instead 
> of _test.py, but I fixed that by adding _tests.py to the list of recognized 
> patterns.
> Get them all running in CircleCI. Nothing special they will be part of the 
> existing dtest jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14526) dtest to validate Cassandra state post failed/successful bootstrap

[Jay Zhuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang updated CASSANDRA-14526:
---
Resolution: Fixed
  Reviewer: Jay Zhuang
Status: Resolved  (was: Patch Available)

> dtest to validate Cassandra state post failed/successful bootstrap
> --
>
> Key: CASSANDRA-14526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14526
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Test/dtest
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
>  Labels: dtest
>
> Please find dtest here:
> || dtest ||
> | [patch 
> |https://github.com/apache/cassandra-dtest/compare/master...jaydeepkumar1984:14526-trunk]|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14931) Backport In-JVM dtests to 2.2, 3.0 and 3.11

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743318#comment-16743318
 ] 

Benedict commented on CASSANDRA-14931:
--

I've force-pushed all branches, which are fresh cherry-picks of all of the 
relevant commits.  It would be great if we could get these in before they go 
stale, although rebasing should be easier now that I have isolated cherry-picks 
for each branch.

I've confirmed no (apparent) leaks on any of the versions, although I have not 
as-yet looked into native memory leaks.

> Backport In-JVM dtests to 2.2, 3.0 and 3.11
> ---
>
> Key: CASSANDRA-14931
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14931
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: Benedict
>Assignee: Benedict
>Priority: Major
> Fix For: 2.2.14, 3.0.18, 3.11.4
>
>
> The In-JVM dtests are of significant value, and much of the testing we are 
> exploring with it can easily be utilised on all presently maintained 
> versions.  We should backport the functionality to at least 3.0.x and 3.11.x 
> - and perhaps even consider 2.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743358#comment-16743358
 ] 

Aleksey Yeschenko commented on CASSANDRA-14958:
---

[~aweisberg] Would've forgotten if not for your comment - thank you. And just 
to confirm, two proper upgrade test runs with the annotation removed, with 
counter tests passing: [3.0|https://circleci.com/gh/iamaleksey/cassandra/921], 
[3.11|https://circleci.com/gh/iamaleksey/cassandra/923].

> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.18, 3.11.4
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster 
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url = 
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
>   ... ;
>  total
> ---
>  0
> (1 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14935) PendingAntiCompaction should be more judicious in the compactions it cancels

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743256#comment-16743256
 ] 

Blake Eggleston commented on CASSANDRA-14935:
-

Looks functionally solid, I like how you collapsed the 2 acquisition attempts 
in {{PendingAntiCompaction}}. There are a few minor / stylistic things though:
 * There’s an unused import in {{AutoSavingCache}}
 * I think {{CancellationType}} can be removed from {{CompactionInfo}}. 
Anywhere it’s set to {{ALL}}, {{sstables.isEmpty()}} should always be true, 
making it redundant.
 * We shouldn't prompt users to check repair_admin in the exception message if 
there are conflicting anti-compactions. Those won’t be caused by hung sessions, 
but users starting multiple repairs at the same time.
 * I think we should use ImmutableSet inside CompactionInfo, but I don’t think 
we should make callers make the immutable copy. If we just accepted a Set and 
made an immutable copy in the ctor, the places we construct CompactionInfo 
would be a bit cleaner.
 * Nit: when quickly reading code, referring to the ActiveCompactionTracker 
instance as {{tracker}} clashes (in my head) with the cfs data tracker. Not a 
huge deal, but calling it something like {{compactions}} or 
{{activeCompactions}} would make it a bit clearer at a glance.

> PendingAntiCompaction should be more judicious in the compactions it cancels
> 
>
> Key: CASSANDRA-14935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14935
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 4.0
>
>
> PendingAntiCompaction currently cancels all ongoing compactions to be able to 
> grab the sstables it requires - it should;
> * avoid stopping ongoing anticompactions - for example, if we have an sstable 
> [0, 100] and are anticompacting it on [0, 50] - now if we start a new 
> incremental repair on [51, 100] we will (try to) stop the first one. Instead 
> we should fail the new incremental repair request.
> * avoid stopping unrelated regular compactions - we should only stop regular 
> compactions for the sstables the new anticompaction needs
> This requires us to keep track of which sstables are being compacted by a 
> given compaction id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14526) dtest to validate Cassandra state post failed/successful bootstrap

[Jay Zhuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743263#comment-16743263
 ] 

Jay Zhuang commented on CASSANDRA-14526:


The new tests passed locally (except: CASSANDRA-14984, not related to this 
change or CASSANDRA-14525). Committed as 
[{{e6f58cb}}|https://github.com/apache/cassandra-dtest/commit/e6f58cb33f7a09f273c5990d5d21c7b529ba80bf].
 Thanks [~chovatia.jayd...@gmail.com].

> dtest to validate Cassandra state post failed/successful bootstrap
> --
>
> Key: CASSANDRA-14526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14526
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Test/dtest
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
>  Labels: dtest
>
> Please find dtest here:
> || dtest ||
> | [patch 
> |https://github.com/apache/cassandra-dtest/compare/master...jaydeepkumar1984:14526-trunk]|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14974) Unit test stdout capture is malfunctioning in 4.0

[Benedict (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benedict updated CASSANDRA-14974:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks, committed as 
[a43b651f8e35dd7081b8593057f118ed0c49cfd6|https://github.com/apache/cassandra/commit/a43b651f8e35dd7081b8593057f118ed0c49cfd6]

> Unit test stdout capture is malfunctioning in 4.0
> -
>
> Key: CASSANDRA-14974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14974
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest, Test/unit
>Reporter: Benedict
>Assignee: Benedict
>Priority: Major
> Fix For: 4.0
>
>
> In 3.x unit tests we make sure to capture stdout to the unit test files, in 
> case tests log to stdout and not to a logger.  However, in 4.0 due to a 
> configuration parameter that is deprecated in logback, the logic is 
> short-circuited silently.
> Once fixed, this affects the cleanup of in-jvm dtests which would register an 
> infinite chain of System.out overrides (one for each node), so we make the 
> functionality robust to multiple instantiations, as well as improve its 
> startup/shutdown sequence guarantees.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743312#comment-16743312
 ] 

Benedict commented on CASSANDRA-14922:
--

Thanks, committed follow-up as 
[00fff3ee6e6c0142529de621bcaeee5790a0c235|https://github.com/apache/cassandra/commit/00fff3ee6e6c0142529de621bcaeee5790a0c235]

> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png, 
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png, 
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png, 
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example 
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1], 
> [example 
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1]) 
> because we use a small container (medium) for unit tests by default and the 
> in JVM dtests are leaking a few hundred megabytes of memory per test right 
> now. This is not a big deal because the dtest runs with the larger containers 
> continue to function fine as well as local testing as the number of in JVM 
> dtests is not yet high enough to cause a problem with more than 2GB of 
> available heap. However we should fix the memory leak so that going forwards 
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be 
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I 
> believe that we have a few potential issues that are leading to the leaks:
> 1. The 
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
>  method is not successfully cleaning up all the metrics created by the 
> {{CassandraMetricsRegistry}}
>  2. The 
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
>  method is not waiting for all the instances to finish shutting down and 
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but 
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
>  does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do 
> not leak memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14985) CircleCI docker image should bake in more dependencies

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14985:
---
Status: Patch Available  (was: Open)

[Docker image 
change|https://github.com/aweisberg/cassandra-builds/commit/e84ed05a971d2bd657a9cc47299229af28d07e2c]
[trunk|https://github.com/aweisberg/cassandra/commit/582c6719d7588fa4bedd71f412118451f2b3ccaf]

dtests:
https://circleci.com/gh/aweisberg/cassandra/2517
https://circleci.com/gh/aweisberg/cassandra/2518
utests:
https://circleci.com/gh/aweisberg/cassandra/2519
upgrade tests:
https://circleci.com/gh/aweisberg/cassandra/2520

>  CircleCI docker image  should bake in more dependencies
> 
>
> Key: CASSANDRA-14985
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14985
> Project: Cassandra
>  Issue Type: Test
>  Components: Test/dtest, Test/unit
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> It's pretty frequent especially in the upgrade tests for either maven or 
> github to fail. I think they are detecting the thundering herd of containers 
> all pulling stuff at the same time and opting out.
> We can reduce this especially for maven dependencies since most are hardly 
> ever changing by downloading everything we can in advance and baking it into 
> the image.
> When building the docker image Initialize the local maven repository by 
> running ant maven-ant-tasks-retrieve-build for 2.1-trunk and do the same with 
> CCM and the Apache repository.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14984) [dtest] 2 TestBootstrap tests failed for branch 2.2

[Jay Zhuang (JIRA)]

Jay Zhuang created CASSANDRA-14984:
--

 Summary: [dtest] 2 TestBootstrap tests failed for branch 2.2
 Key: CASSANDRA-14984
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14984
 Project: Cassandra
  Issue Type: Bug
  Components: Test/dtest
Reporter: Jay Zhuang


Failed tests:
{noformat}
test_decommissioned_wiped_node_can_join
test_decommissioned_wiped_node_can_gossip_to_single_seed
{noformat}

Error:
{noformat}
...
# Decommision the new node and kill it
logger.debug("Decommissioning & stopping node2")
>   node2.decommission()
...
def handle_external_tool_process(process, cmd_args):
out, err = process.communicate()
if (out is not None) and isinstance(out, bytes):
out = out.decode()
if (err is not None) and isinstance(err, bytes):
err = err.decode()
rc = process.returncode

if rc != 0:
>   raise ToolError(cmd_args, rc, out, err)
E   ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', 
'-p', '7200', 'decommission'] exited with non-zero status; exit status: 2;
E   stderr: error: Thread signal failed
E   -- StackTrace --
E   java.io.IOException: Thread signal failed
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14951) Add a script to make running the cqlsh tests in cassandra repo easier

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14951:
---
Fix Version/s: 4.0.x
   3.11.x
   3.0.x

> Add a script to make running the cqlsh tests in cassandra repo easier
> -
>
> Key: CASSANDRA-14951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14951
> Project: Cassandra
>  Issue Type: Test
>  Components: Tool/cqlsh
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Some CQLSH tests have not been running. They need to be reenabled and fixed 
> before 4.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14985) CircleCI docker image should bake in more dependencies

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743330#comment-16743330
 ] 

Ariel Weisberg edited comment on CASSANDRA-14985 at 1/15/19 9:15 PM:
-

[Docker image 
change|https://github.com/aweisberg/cassandra-builds/commit/cc637ecf848769c4949414e7bc97ce9e90dcbfd4]
[trunk|https://github.com/aweisberg/cassandra/commit/582c6719d7588fa4bedd71f412118451f2b3ccaf]

dtests:
https://circleci.com/gh/aweisberg/cassandra/2517
https://circleci.com/gh/aweisberg/cassandra/2518
utests:
https://circleci.com/gh/aweisberg/cassandra/2519
upgrade tests:
https://circleci.com/gh/aweisberg/cassandra/2520


was (Author: aweisberg):
[Docker image 
change|https://github.com/aweisberg/cassandra-builds/commit/e84ed05a971d2bd657a9cc47299229af28d07e2c]
[trunk|https://github.com/aweisberg/cassandra/commit/582c6719d7588fa4bedd71f412118451f2b3ccaf]

dtests:
https://circleci.com/gh/aweisberg/cassandra/2517
https://circleci.com/gh/aweisberg/cassandra/2518
utests:
https://circleci.com/gh/aweisberg/cassandra/2519
upgrade tests:
https://circleci.com/gh/aweisberg/cassandra/2520

>  CircleCI docker image  should bake in more dependencies
> 
>
> Key: CASSANDRA-14985
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14985
> Project: Cassandra
>  Issue Type: Test
>  Components: Test/dtest, Test/unit
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> It's pretty frequent especially in the upgrade tests for either maven or 
> github to fail. I think they are detecting the thundering herd of containers 
> all pulling stuff at the same time and opting out.
> We can reduce this especially for maven dependencies since most are hardly 
> ever changing by downloading everything we can in advance and baking it into 
> the image.
> When building the docker image Initialize the local maven repository by 
> running ant maven-ant-tasks-retrieve-build for 2.1-trunk and do the same with 
> CCM and the Apache repository.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-14920) Some comparisons used for verifying paging queries in dtests only test the column names and not values

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg resolved CASSANDRA-14920.

Resolution: Fixed

> Some comparisons used for verifying paging queries in dtests only test the 
> column names and not values
> --
>
> Key: CASSANDRA-14920
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14920
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest
>Reporter: Sam Tunnicliffe
>Assignee: Ariel Weisberg
>Priority: Major
>
> The implementation of {{PageAssertionMixin::assertEqualIgnoreOrder}} 
> introduced in CASSANDRA-14134 can't be used to compare expected and actual 
> results when the row data is represented by a {{dict}}. The underlying 
> {{list_to_hashed_dict}} function expected lists of values and when it 
> encounters a dict, it constructs its normalized list using only the keys. So 
> the actual result values may be completely wrong, but as long as the field 
> names are the same the equality check will pass. This affects only 
> {{paging_tests.py}} and {{upgrade_tests/paging_test.py}}, and looks like it 
> maybe a leftover from an earlier dev iteration, as some tests in the affected 
> fixtures are already using the alternative (and correct) 
> {{assertions.py::assert_lists_equal_ignoring_order}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14986) JVM Crashes with SimpleComposite.size

[Ian Spence (JIRA)]

Ian Spence created CASSANDRA-14986:
--

 Summary: JVM Crashes with SimpleComposite.size
 Key: CASSANDRA-14986
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14986
 Project: Cassandra
  Issue Type: Bug
 Environment: Cassandra 2.2.13
 2 DCs. 3 RACs in DC1, 6 RACs in DC2.
 CentOS 6.9 (server that crashed) and CentOS 7
 JRE 1.8.0_74
Reporter: Ian Spence
 Attachments: hserrpid13058.log

The JVM for Cassandra crashed on a running node in a multi-DC cluster. Looking 
at the dump, it crashed on a method that absolutely should not have triggered a 
fault like this.

A sanitized version of the dump is attached. It appears to be crashing in this 
file: 
[https://raw.githubusercontent.com/apache/cassandra/cassandra-2.2/src/java/org/apache/cassandra/db/composites/SimpleComposite.java]

Specifically at:
{noformat}
public int size()
{
return 1;
}{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14920) Some comparisons used for verifying paging queries in dtests only test the column names and not values

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743414#comment-16743414
 ] 

Ariel Weisberg commented on CASSANDRA-14920:


I resolved this as part of 14421, I modifed 
https://github.com/apache/cassandra-dtest/commit/84598f11513f4c1dc0be4d7115a47b59940a649e#diff-96abe2232f1118eb0579d88abe504a9eL167
 a bit and also pointed some tests at the other equality method that creates an 
order by sorting on a key.

> Some comparisons used for verifying paging queries in dtests only test the 
> column names and not values
> --
>
> Key: CASSANDRA-14920
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14920
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest
>Reporter: Sam Tunnicliffe
>Assignee: Ariel Weisberg
>Priority: Major
>
> The implementation of {{PageAssertionMixin::assertEqualIgnoreOrder}} 
> introduced in CASSANDRA-14134 can't be used to compare expected and actual 
> results when the row data is represented by a {{dict}}. The underlying 
> {{list_to_hashed_dict}} function expected lists of values and when it 
> encounters a dict, it constructs its normalized list using only the keys. So 
> the actual result values may be completely wrong, but as long as the field 
> names are the same the equality check will pass. This affects only 
> {{paging_tests.py}} and {{upgrade_tests/paging_test.py}}, and looks like it 
> maybe a leftover from an earlier dev iteration, as some tests in the affected 
> fixtures are already using the alternative (and correct) 
> {{assertions.py::assert_lists_equal_ignoring_order}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14951) Add a script to make running the cqlsh tests in cassandra repo easier

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14951:
---
Issue Type: Test  (was: Bug)

> Add a script to make running the cqlsh tests in cassandra repo easier
> -
>
> Key: CASSANDRA-14951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14951
> Project: Cassandra
>  Issue Type: Test
>  Components: Tool/cqlsh
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> Some CQLSH tests have not been running. They need to be reenabled and fixed 
> before 4.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14951) Add a script to make running the cqlsh tests in cassandra repo easier

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14951:
---
Summary: Add a script to make running the cqlsh tests in cassandra repo 
easier  (was: Reenable CQLSH tests)

> Add a script to make running the cqlsh tests in cassandra repo easier
> -
>
> Key: CASSANDRA-14951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/cqlsh
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> Some CQLSH tests have not been running. They need to be reenabled and fixed 
> before 4.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11323) When node runs out of commitlog space you get poor log information

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743416#comment-16743416
 ] 

Ariel Weisberg commented on CASSANDRA-11323:


[~BorisO] are you going to be able to come back to this?

> When node runs out of commitlog space you get poor log information
> --
>
> Key: CASSANDRA-11323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11323
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Observability
>Reporter: T Jake Luciani
>Assignee: Boris Onufriyev
>Priority: Trivial
>  Labels: fallout
> Attachments: 11323-2.2.txt, 11323-3.0.txt, 11323-3.11.txt, 
> 11323-trunk.txt
>
>
> {code}
> ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
> StorageService.java:470 - Stopping gossiper
> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
> StorageService.java:377 - Stopping gossip by operator request
> INFO  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 Gossiper.java:1463 
> - Announcing shutdown
> {code}
> That's all you get when a node runs out of commit log space. 
> We should explicitly callout the fact the commitlog is out of disk.  I see 
> that in the commit log error handler but after it shuts down. So I think it's 
> never getting written before shutdown.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14986) JVM Crashes with SimpleComposite.size

[Dinesh Joshi (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743449#comment-16743449
 ] 

Dinesh Joshi commented on CASSANDRA-14986:
--

Hi [~ecnepsnai], thanks for reporting this issue. Could you add any more 
details that will help reproduce this issue locally?

> JVM Crashes with SimpleComposite.size
> -
>
> Key: CASSANDRA-14986
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14986
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.2.13
>  2 DCs. 3 RACs in DC1, 6 RACs in DC2.
>  CentOS 6.9 (server that crashed) and CentOS 7
>  JRE 1.8.0_74
>Reporter: Ian Spence
>Priority: Major
> Attachments: hserrpid13058.log
>
>
> The JVM for Cassandra crashed on a running node in a multi-DC cluster. 
> Looking at the dump, it crashed on a method that absolutely should not have 
> triggered a fault like this.
> A sanitized version of the dump is attached. It appears to be crashing in 
> this file: 
> [https://raw.githubusercontent.com/apache/cassandra/cassandra-2.2/src/java/org/apache/cassandra/db/composites/SimpleComposite.java]
> Specifically at:
> {noformat}
> public int size()
> {
> return 1;
> }{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (a43b651 -> c291e8d)

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from a43b651  Unit test stdout capture is malfunctioning in 4.0
 new a9d1af9  Add a script to make running the cqlsh tests in cassandra 
repo easier
 new 25b72d0  Merge branch '14951-3.0' into 14951-3.11
 new c291e8d  Merge branch '14951-3.11' into 14951-trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 pylib/cassandra-cqlsh-tests.sh | 100 +
 pylib/cqlshlib/setup.cfg   |   4 ++
 pylib/requirements.txt |  21 +
 4 files changed, 126 insertions(+)
 create mode 100755 pylib/cassandra-cqlsh-tests.sh
 create mode 100644 pylib/cqlshlib/setup.cfg
 create mode 100644 pylib/requirements.txt


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.11 updated (62f0280 -> 25b72d0)

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 62f0280  Merge branch 'cassandra-3.0' into cassandra-3.11
 new a9d1af9  Add a script to make running the cqlsh tests in cassandra 
repo easier
 new 25b72d0  Merge branch '14951-3.0' into 14951-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 pylib/cassandra-cqlsh-tests.sh | 100 +
 pylib/cqlshlib/setup.cfg   |   4 ++
 pylib/requirements.txt |  21 +
 4 files changed, 126 insertions(+)
 create mode 100755 pylib/cassandra-cqlsh-tests.sh
 create mode 100644 pylib/cqlshlib/setup.cfg
 create mode 100644 pylib/requirements.txt


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14951) Add a script to make running the cqlsh tests in cassandra repo easier

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14951:
---
Status: Ready to Commit  (was: Patch Available)

> Add a script to make running the cqlsh tests in cassandra repo easier
> -
>
> Key: CASSANDRA-14951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14951
> Project: Cassandra
>  Issue Type: Test
>  Components: Tool/cqlsh
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Some CQLSH tests have not been running. They need to be reenabled and fixed 
> before 4.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14957) Rolling Restart Of Nodes Causes Dataloss Due To Schema Collision

[Vincent White (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741791#comment-16741791
 ] 

Vincent White edited comment on CASSANDRA-14957 at 1/16/19 4:14 AM:


I believe this can happen due a race condition when issuing create statements. 
If a CREATE TABLE statement for the same name is issued -on a two different 
nodes- at the same time you then series of events will look like the following:

Node1 will create and propagate the table as normal, with column family ID 
cf_id1.

If the Node2 gets past the below check before receiving the schema change from 
the Node1, then Node2 will continue executing its CREATE TABLE statement as 
normal except with its own column family ID cf_id2.
{code:java|title=org/apache/cassandra/service/MigrationManager.java:378}
   // If we have a table or a view which has the same name, we can't add a 
new one
else if (throwOnDuplicate && ksm.getTableOrViewNullable(cfm.cfName) != 
null)
throw new AlreadyExistsException(cfm.ksName, cfm.cfName);

logger.info("Create new table: {}", cfm);
announce(SchemaKeyspace.makeCreateTableMutation(ksm, cfm, timestamp), 
announceLocally);
{code}
Node2 will send out its own set of schema mutations as normal via announce(). 
On all nodes that receive this change, and locally on Node2, they will write 
the schema changes to disk 
(*org/apache/cassandra/schema/SchemaKeyspace.java:1390)* before attempting to 
merge them with their live schema. When attempting to merge the changes with 
their live schema 
*org.apache.cassandra.config.CFMetaData#validateCompatibility* will throw a 
configuration exception and stop the new change being merged. The changes 
written to disk are not rolled back.

All nodes will continue to use the table definition in their live schema 
(cf_id1) and everything will continue to work as expected as if the second 
CREATE TABLE statement was ignored. The issue is that all nodes now have the 
wrong column family ID recorded in their *system_schema.tables* system tables. 
When the nodes restart they we read their schema from disk and start using the 
wrong column family ID, at which point they will make a new empty folder on 
disk for it and you will start seeing the types of errors you've mentioned.

This of course isn't just limited to corrupting the column family ID but I 
believe this can apply to any part of the column family definition.

I believe this is solved in trunk with the changes introduced as part of 
CASSANDRA-10699

EDIT: This doesn't require the statements going to two different nodes, it can 
happen with both statements going to the same nodes.


was (Author: vincentwhite):
I believe this can happen due a race condition when issuing create statements. 
If a CREATE TABLE statement for the same name is issued on a two different 
nodes at the same time you then series of events will look like the following:

Node1 will create and propagate the table as normal, with column family ID 
cf_id1.

If the Node2 gets past the below check before receiving the schema change from 
the Node1, then Node2 will continue executing its CREATE TABLE statement as 
normal except with its own column family ID cf_id2.
{code:java|title=org/apache/cassandra/service/MigrationManager.java:378}
   // If we have a table or a view which has the same name, we can't add a 
new one
else if (throwOnDuplicate && ksm.getTableOrViewNullable(cfm.cfName) != 
null)
throw new AlreadyExistsException(cfm.ksName, cfm.cfName);

logger.info("Create new table: {}", cfm);
announce(SchemaKeyspace.makeCreateTableMutation(ksm, cfm, timestamp), 
announceLocally);
{code}
Node2 will send out its own set of schema mutations as normal via announce(). 
On all nodes that receive this change, and locally on Node2, they will write 
the schema changes to disk 
(*org/apache/cassandra/schema/SchemaKeyspace.java:1390)* before attempting to 
merge them with their live schema. When attempting to merge the changes with 
their live schema 
*org.apache.cassandra.config.CFMetaData#validateCompatibility* will throw a 
configuration exception and stop the new change being merged. The changes 
written to disk are not rolled back.

All nodes will continue to use the table definition in their live schema 
(cf_id1) and everything will continue to work as expected as if the second 
CREATE TABLE statement was ignored. The issue is that all nodes now have the 
wrong column family ID recorded in their *system_schema.tables* system tables. 
When the nodes restart they we read their schema from disk and start using the 
wrong column family ID, at which point they will make a new empty folder on 
disk for it and you will start seeing the types of errors you've mentioned.

This of course isn't just limited to corrupting the column family ID but I 
believe this can apply to any 

[jira] [Commented] (CASSANDRA-14957) Rolling Restart Of Nodes Causes Dataloss Due To Schema Collision

[Vincent White (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743600#comment-16743600
 ] 

Vincent White commented on CASSANDRA-14957:
---

 

This doesn't require a new CREATE TABLE statement to be issued while restarting 
the nodes etc, only that at the time of the table's creation, somehow two 
CREATE TABLES statements were issued for the same table.

Slightly unrelated: I do wonder if there is a driver somewhere out there that 
was accidentally applying something like speculative execution to DDL 
statements because I do see this issue in situations where it would be unlikely 
that the operator was manually sending off two CREATE TABLE statements so close 
together.

Where the node restarts come into play is that this condition will go unnoticed 
until the nodes are restarted and they read the "wrong"/new table ID from the 
system_schema tables. Since table ID's are timeuuid's we see that both the old 
and new ID's of your tasks table were originally generated on February 19, 2018 
11:26:33 AM GMT. This would indicate that these nodes haven't been restarted 
since then? (assuming no drastic change in your computer clock)

 

> Rolling Restart Of Nodes Causes Dataloss Due To Schema Collision
> 
>
> Key: CASSANDRA-14957
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14957
> Project: Cassandra
>  Issue Type: Bug
>  Components: Cluster/Schema
>Reporter: Avraham Kalvo
>Priority: Major
>
> We were issuing a rolling restart on a mission-critical five node C* cluster.
> The first node which was restarted got the following messages in its 
> system.log:
> ```
> January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing 
> tasks_scheduler_external.tasks
> ```
> ```
> WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
> cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this 
> is likely due to the schema not being fully propagated. Please wait for 
> schema agreement on table creation.
> at 
> org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> ```
> The latter was then repeated several times across the cluster.
> It was then found out that the table in question 
> `tasks_scheduler_external.tasks` was created with a new schema version 
> sometime along the entire cluster consecutive restart and became available 
> once the schema agreement settled, which started taking requests leaving the 
> previous version of the schema unavailable for any request, thus generating a 
> data loss to our online system.
> Data loss was recovered by manually copying SSTables from the previous 
> version directory of the schema to the new one followed by `nodetool refresh` 
> to the relevant table.
> The above has repeated itself for several tables across various keyspaces.
> One other thing to mention is that a repair was in place for the first node 
> to be restarted, which was obviously stopped as the daemon was shut down, but 
> this doesn't seem to do with the above at first glance.
> Seems somewhat related to:
> https://issues.apache.org/jira/browse/CASSANDRA-13559



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch '14951-3.0' into 14951-3.11

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 25b72d08c000d6c09d76b2b117d79cb504d14f1a
Merge: 62f0280 a9d1af9
Author: Ariel Weisberg 
AuthorDate: Tue Jan 15 18:08:18 2019 -0500

Merge branch '14951-3.0' into 14951-3.11

 CHANGES.txt|   1 +
 pylib/cassandra-cqlsh-tests.sh | 100 +
 pylib/cqlshlib/setup.cfg   |   4 ++
 pylib/requirements.txt |  21 +
 4 files changed, 126 insertions(+)

diff --cc CHANGES.txt
index 477afd8,5faac66..009e8d4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.18
 +3.11.4
 + * Make stop-server.bat wait for Cassandra to terminate (CASSANDRA-14829)
 + * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
 +Merged from 3.0:
+  * Add a script to make running the cqlsh tests in cassandra repo easier 
(CASSANDRA-14951)
   * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
   * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
   * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.0 updated: Add a script to make running the cqlsh tests in cassandra repo easier

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new a9d1af9  Add a script to make running the cqlsh tests in cassandra 
repo easier
a9d1af9 is described below

commit a9d1af96b515c06f56b49fd9894c3f7216aa099d
Author: Dinesh A. Joshi 
AuthorDate: Fri Jan 4 12:43:34 2019 -0800

Add a script to make running the cqlsh tests in cassandra repo easier

Patch by Dinesh Joshi; Reviewed by Ariel Weisberg for CASSANDRA-14951
---
 CHANGES.txt|   1 +
 pylib/cassandra-cqlsh-tests.sh | 100 +
 pylib/cqlshlib/setup.cfg   |   4 ++
 pylib/requirements.txt |  21 +
 4 files changed, 126 insertions(+)

diff --git a/CHANGES.txt b/CHANGES.txt
index bb8b54c..5faac66 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.18
+ * Add a script to make running the cqlsh tests in cassandra repo easier 
(CASSANDRA-14951)
  * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
  * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
  * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)
diff --git a/pylib/cassandra-cqlsh-tests.sh b/pylib/cassandra-cqlsh-tests.sh
new file mode 100755
index 000..1fb3aa0
--- /dev/null
+++ b/pylib/cassandra-cqlsh-tests.sh
@@ -0,0 +1,100 @@
+#!/bin/bash -x
+
+
+#
+# Prep
+#
+
+
+WORKSPACE=$1
+
+if [ "${WORKSPACE}" = "" ]; then
+echo "Specify Cassandra source directory"
+exit
+fi
+
+export PYTHONIOENCODING="utf-8"
+export PYTHONUNBUFFERED=true
+export CASS_DRIVER_NO_EXTENSIONS=true
+export CASS_DRIVER_NO_CYTHON=true
+export CCM_MAX_HEAP_SIZE="2048M"
+export CCM_HEAP_NEWSIZE="200M"
+export CCM_CONFIG_DIR=${WORKSPACE}/.ccm
+export NUM_TOKENS="32"
+export CASSANDRA_DIR=${WORKSPACE}
+
+# Loop to prevent failure due to maven-ant-tasks not downloading a jar..
+for x in $(seq 1 3); do
+ant -buildfile ${CASSANDRA_DIR}/build.xml realclean jar
+RETURN="$?"
+if [ "${RETURN}" -eq "0" ]; then
+break
+fi
+done
+# Exit, if we didn't build successfully
+if [ "${RETURN}" -ne "0" ]; then
+echo "Build failed with exit code: ${RETURN}"
+exit ${RETURN}
+fi
+
+# Set up venv with dtest dependencies
+set -e # enable immediate exit if venv setup fails
+virtualenv --python=python2 --no-site-packages venv
+source venv/bin/activate
+pip install -r ${CASSANDRA_DIR}/pylib/requirements.txt
+pip freeze
+
+if [ "$cython" = "yes" ]; then
+pip install "Cython>=0.20,<0.25"
+cd pylib/; python setup.py build_ext --inplace
+cd ${WORKSPACE}
+fi
+
+
+#
+# Main
+#
+
+
+ccm remove test || true # in case an old ccm cluster is left behind
+ccm create test -n 1 --install-dir=${CASSANDRA_DIR}
+ccm updateconf "enable_user_defined_functions: true"
+
+version_from_build=$(ccm node1 versionfrombuild)
+export pre_or_post_cdc=$(python -c """from distutils.version import 
LooseVersion
+print \"postcdc\" if LooseVersion(\"${version_from_build}\") >= \"3.8\" else 
\"precdc\"
+""")
+case "${pre_or_post_cdc}" in
+postcdc)
+ccm updateconf "cdc_enabled: true"
+;;
+precdc)
+:
+;;
+*)
+echo "${pre_or_post_cdc}" is an invalid value.
+exit 1
+;;
+esac
+
+ccm start --wait-for-binary-proto
+
+cd ${CASSANDRA_DIR}/pylib/cqlshlib/
+
+set +e # disable immediate exit from this point
+nosetests
+
+ccm remove
+mv nosetests.xml ${WORKSPACE}/cqlshlib.xml
+
+
+#
+# Clean
+#
+
+
+# /virtualenv
+deactivate
+
+# Exit cleanly for usable "Unstable" status
+exit 0
diff --git a/pylib/cqlshlib/setup.cfg b/pylib/cqlshlib/setup.cfg
new file mode 100644
index 000..6c523ee
--- /dev/null
+++ b/pylib/cqlshlib/setup.cfg
@@ -0,0 +1,4 @@
+[nosetests]
+verbosity=3
+detailed-errors=1
+with-xunit=1
diff --git a/pylib/requirements.txt b/pylib/requirements.txt
new file mode 100644
index 000..a9b6217
--- /dev/null
+++ b/pylib/requirements.txt
@@ -0,0 +1,21 @@
+# See python driver docs: futures and six have to be installed before
+# cythonizing the driver, perhaps only on old pips.
+# 
http://datastax.github.io/python-driver/installation.html#cython-based-extensions
+futures
+six
+-e 
git+https://github.com/datastax/python-driver.git@cassandra-test#egg=cassandra-driver
+# Used ccm version is tracked by cassandra-test branch in ccm repo. Please 
create a PR there for fixes or upgrades to new releases.
+-e git+https://github.com/riptano/ccm.git@cassandra-test#egg=ccm
+cql
+decorator
+docopt
+enum34
+flaky
+mock
+nose

[cassandra] 01/01: Merge branch '14951-3.11' into 14951-trunk

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit c291e8d81585d64e6d14349ca512bf0048689e91
Merge: a43b651 25b72d0
Author: Ariel Weisberg 
AuthorDate: Tue Jan 15 18:08:31 2019 -0500

Merge branch '14951-3.11' into 14951-trunk

 CHANGES.txt|   1 +
 pylib/cassandra-cqlsh-tests.sh | 100 +
 pylib/cqlshlib/setup.cfg   |   4 ++
 pylib/requirements.txt |  21 +
 4 files changed, 126 insertions(+)

diff --cc CHANGES.txt
index 5af5b64,009e8d4..0e1da11
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -343,9 -2,12 +343,10 @@@
   * Make stop-server.bat wait for Cassandra to terminate (CASSANDRA-14829)
   * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
  Merged from 3.0:
+  * Add a script to make running the cqlsh tests in cassandra repo easier 
(CASSANDRA-14951)
   * If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table. (CASSANDRA-14905)
 - * Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters 
(CASSANDRA-14958)
   * Streaming needs to synchronise access to LifecycleTransaction 
(CASSANDRA-14554)
   * Fix cassandra-stress write hang with default options (CASSANDRA-14616)
 - * Differentiate between slices and RTs when decoding legacy bounds 
(CASSANDRA-14919)
   * Netty epoll IOExceptions caused by unclean client disconnects being logged 
at INFO (CASSANDRA-14909)
   * Unfiltered.isEmpty conflicts with Row extends AbstractCollection.isEmpty 
(CASSANDRA-14588)
   * RangeTombstoneList doesn't properly clean up mergeable or superseded rts 
in some cases (CASSANDRA-14894)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14951) Add a script to make running the cqlsh tests in cassandra repo easier

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14951:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Committed as 
[a9d1af96b515c06f56b49fd9894c3f7216aa099d|https://github.com/apache/cassandra/commit/a9d1af96b515c06f56b49fd9894c3f7216aa099d].
 Thanks!

Tests 
https://circleci.com/gh/aweisberg/cassandra/tree/14951-trunk
https://circleci.com/gh/aweisberg/cassandra/tree/14951-3.11
https://circleci.com/gh/aweisberg/cassandra/tree/14951-3.0

> Add a script to make running the cqlsh tests in cassandra repo easier
> -
>
> Key: CASSANDRA-14951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14951
> Project: Cassandra
>  Issue Type: Test
>  Components: Tool/cqlsh
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Some CQLSH tests have not been running. They need to be reenabled and fixed 
> before 4.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14957) Rolling Restart Of Nodes Causes Dataloss Due To Schema Collision

[Avraham Kalvo (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742861#comment-16742861
 ] 

Avraham Kalvo commented on CASSANDRA-14957:
---

Thanks for the elaborated explanation [~VincentWhite],

Can you think of a condition where the above is experienced without an explicit 
CREATE TABLE being issued? (mind you the table was there before the rolling 
restart, and by the end of the rolling restart it became available with the new 
version) 

As all was done was merely to restart the nodes one by one, no CREATE TABLE 
statements were issued throughout that restart whatsoever, The schema was 
settled across all cluster eventually, as all of the nodes were restarted 
consecutively.

Avi.


> Rolling Restart Of Nodes Causes Dataloss Due To Schema Collision
> 
>
> Key: CASSANDRA-14957
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14957
> Project: Cassandra
>  Issue Type: Bug
>  Components: Cluster/Schema
>Reporter: Avraham Kalvo
>Priority: Major
>
> We were issuing a rolling restart on a mission-critical five node C* cluster.
> The first node which was restarted got the following messages in its 
> system.log:
> ```
> January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing 
> tasks_scheduler_external.tasks
> ```
> ```
> WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
> cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this 
> is likely due to the schema not being fully propagated. Please wait for 
> schema agreement on table creation.
> at 
> org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> ```
> The latter was then repeated several times across the cluster.
> It was then found out that the table in question 
> `tasks_scheduler_external.tasks` was created with a new schema version 
> sometime along the entire cluster consecutive restart and became available 
> once the schema agreement settled, which started taking requests leaving the 
> previous version of the schema unavailable for any request, thus generating a 
> data loss to our online system.
> Data loss was recovered by manually copying SSTables from the previous 
> version directory of the schema to the new one followed by `nodetool refresh` 
> to the relevant table.
> The above has repeated itself for several tables across various keyspaces.
> One other thing to mention is that a repair was in place for the first node 
> to be restarted, which was obviously stopped as the daemon was shut down, but 
> this doesn't seem to do with the above at first glance.
> Seems somewhat related to:
> https://issues.apache.org/jira/browse/CASSANDRA-13559



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14881) Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 3.11.2

[Laxmikant Upadhyay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laxmikant Upadhyay updated CASSANDRA-14881:
---
Attachment: (was: CorrectCounterUpdateTrace.txt)

> Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 
> 3.11.2
> --
>
> Key: CASSANDRA-14881
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14881
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Core
> Environment: {code:java}
> // code placeholder
> {code}
>Reporter: Laxmikant Upadhyay
>Priority: Major
>
> I have 3 node (A,B,C) 2.1.16 cassandra cluster which i am upgrading to 
> cassandra 3.11.2.
>  My current cluster status is node a has been upgrade to 3.11.2, B is down, 
> and C is on cassandra 2.1.16
> when i run counter update using cqlsh it is behaving strange inconsistent way 
> , sometimes the update actually applied and sometimes it does not apply at 
> all.
> See the below example of cqlsh logged in to node A:
>  ===Incorrect update : update not applied 
> {code:java}
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0
>  
> user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 
> 'key1' AND column1 = 'count';
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0{code}
>  ===Correct update : update applied successfully 
> {code:java}
> user@cqlsh> USE ks ;
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count |  -100
>  
> user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 
> 'key1' AND column1 = 'count';
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0{code}
> I have attached the output with trace enabled (with actual timestamp) for 
> both correct and incorrect counter update .
> What is the reason of this weird behavior ?
> Below is my table details:
> {code:java}
> CREATE TABLE ks."counterTable" (
>     key text,
>     column1 text,
>     value counter,
>     PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE
>     AND CLUSTERING ORDER BY (column1 ASC)
>     AND bloom_filter_fp_chance = 0.01
>     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>     AND comment = ''
>     AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
>     AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND crc_check_chance = 1.0
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 0
>     AND gc_grace_seconds = 864000
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99PERCENTILE';{code}
> *Note :* When all nodes are successfully upgraded OR when 2 nodes gets 
> upgraded to 3.11.2 and 3rd non-upgraded node is down then we don't see such 
> issue . couner update works as expected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14881) Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 3.11.2

[Laxmikant Upadhyay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laxmikant Upadhyay updated CASSANDRA-14881:
---
Attachment: (was: IncorrectCounterUpdateTrace.txt)

> Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 
> 3.11.2
> --
>
> Key: CASSANDRA-14881
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14881
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Core
> Environment: {code:java}
> // code placeholder
> {code}
>Reporter: Laxmikant Upadhyay
>Priority: Major
>
> I have 3 node (A,B,C) 2.1.16 cassandra cluster which i am upgrading to 
> cassandra 3.11.2.
>  My current cluster status is node a has been upgrade to 3.11.2, B is down, 
> and C is on cassandra 2.1.16
> when i run counter update using cqlsh it is behaving strange inconsistent way 
> , sometimes the update actually applied and sometimes it does not apply at 
> all.
> See the below example of cqlsh logged in to node A:
>  ===Incorrect update : update not applied 
> {code:java}
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0
>  
> user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 
> 'key1' AND column1 = 'count';
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0{code}
>  ===Correct update : update applied successfully 
> {code:java}
> user@cqlsh> USE ks ;
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count |  -100
>  
> user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 
> 'key1' AND column1 = 'count';
> user@cqlsh:ks> select * from "counterTable";
>  key  | column1    | value
> --++---
>  key1 | count | 0{code}
> I have attached the output with trace enabled (with actual timestamp) for 
> both correct and incorrect counter update .
> What is the reason of this weird behavior ?
> Below is my table details:
> {code:java}
> CREATE TABLE ks."counterTable" (
>     key text,
>     column1 text,
>     value counter,
>     PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE
>     AND CLUSTERING ORDER BY (column1 ASC)
>     AND bloom_filter_fp_chance = 0.01
>     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>     AND comment = ''
>     AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
>     AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND crc_check_chance = 1.0
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 0
>     AND gc_grace_seconds = 864000
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99PERCENTILE';{code}
> *Note :* When all nodes are successfully upgraded OR when 2 nodes gets 
> upgraded to 3.11.2 and 3rd non-upgraded node is down then we don't see such 
> issue . couner update works as expected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14935) PendingAntiCompaction should be more judicious in the compactions it cancels

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14935:

Reviewer: Blake Eggleston
  Status: Patch Available  (was: Open)

tests: https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F14935
patch: 
https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14935?expand=1

basic idea is to track which sstables are involved in the compaction in 
{{CompactionInfo}} and give {{runWithCompactionsDisabled}} a predicate on which 
sstables to abort compaction for.


> PendingAntiCompaction should be more judicious in the compactions it cancels
> 
>
> Key: CASSANDRA-14935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14935
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 4.0
>
>
> PendingAntiCompaction currently cancels all ongoing compactions to be able to 
> grab the sstables it requires - it should;
> * avoid stopping ongoing anticompactions - for example, if we have an sstable 
> [0, 100] and are anticompacting it on [0, 50] - now if we start a new 
> incremental repair on [51, 100] we will (try to) stop the first one. Instead 
> we should fail the new incremental repair request.
> * avoid stopping unrelated regular compactions - we should only stop regular 
> compactions for the sstables the new anticompaction needs
> This requires us to keep track of which sstables are being compacted by a 
> given compaction id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14935) PendingAntiCompaction should be more judicious in the compactions it cancels

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742884#comment-16742884
 ] 

Marcus Eriksson edited comment on CASSANDRA-14935 at 1/15/19 9:17 AM:
--

tests: https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F14935
patch: 
https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14935?expand=1

Basic idea is to track which sstables are involved in which compaction in 
{{CompactionInfo}} and give {{runWithCompactionsDisabled}} a predicate on which 
sstables to abort compaction for. The predicate is then used when starting 
anticompactions (both to avoid cancelling compactions in repaired data and for 
the non-intersecting ranges) and when starting sub range compactions


was (Author: krummas):
tests: https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F14935
patch: 
https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14935?expand=1

basic idea is to track which sstables are involved in the compaction in 
{{CompactionInfo}} and give {{runWithCompactionsDisabled}} a predicate on which 
sstables to abort compaction for.


> PendingAntiCompaction should be more judicious in the compactions it cancels
> 
>
> Key: CASSANDRA-14935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14935
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 4.0
>
>
> PendingAntiCompaction currently cancels all ongoing compactions to be able to 
> grab the sstables it requires - it should;
> * avoid stopping ongoing anticompactions - for example, if we have an sstable 
> [0, 100] and are anticompacting it on [0, 50] - now if we start a new 
> incremental repair on [51, 100] we will (try to) stop the first one. Instead 
> we should fail the new incremental repair request.
> * avoid stopping unrelated regular compactions - we should only stop regular 
> compactions for the sstables the new anticompaction needs
> This requires us to keep track of which sstables are being compacted by a 
> given compaction id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch master updated: If SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table

[aweisberg]

This is an automated email from the ASF dual-hosted git repository.

aweisberg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new 7a6d900  If SizeEstimatesRecorder misses a 'onDropTable' notification, 
the size_estimates table will never be cleared for that table
7a6d900 is described below

commit 7a6d9002709628de2bc6af9d987a189b302e4472
Author: Joel Knighton 
AuthorDate: Tue Sep 12 17:33:56 2017 -0500

If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
size_estimates table will never be cleared for that table

Patch by Joel Knighton; Reviewed by Ariel Weisberg for CASSANDRA-14905

Co-authored-by: Joel Knighton 
Co-authored-by: Aleksandr Sorokoumov 
---
 nodetool_test.py| 20 ++
 schema_test.py  | 61 +++--
 tools/assertions.py | 19 +++--
 3 files changed, 96 insertions(+), 4 deletions(-)

diff --git a/nodetool_test.py b/nodetool_test.py
index fb73d30..012780b 100644
--- a/nodetool_test.py
+++ b/nodetool_test.py
@@ -300,6 +300,26 @@ class TestNodetool(Tester):
 session.execute('INSERT INTO test.test (pk, ck, val) VALUES (0, 1, 
2);')
 assert_all(session, 'SELECT pk, ck, val FROM test.test;', [[0, 1, 2]])
 
+@since('3.0')
+def test_refresh_size_estimates_clears_invalid_entries(self):
+"""
+@jira_ticket CASSANDRA-14905
+ nodetool refreshsizeestimates should clear up entries for tables that 
no longer exist
+"""
+cluster = self.cluster
+cluster.populate(1)
+node = cluster.nodelist()[0]
+cluster.start()
+session = self.patient_exclusive_cql_connection(node)
+session.execute("USE system;")
+# Valid keyspace but invalid table
+session.execute("INSERT INTO size_estimates (keyspace_name, 
table_name, range_start, range_end, mean_partition_size, partitions_count) 
VALUES ('system_auth', 'bad_table', '-5', '5', 0, 0);")
+# Invalid keyspace and table
+session.execute("INSERT INTO size_estimates (keyspace_name, 
table_name, range_start, range_end, mean_partition_size, partitions_count) 
VALUES ('bad_keyspace', 'bad_table', '-5', '5', 0, 0);")
+node.nodetool('refreshsizeestimates')
+assert_none(session, "SELECT * FROM size_estimates WHERE 
keyspace_name='system_auth' AND table_name='bad_table'")
+assert_none(session, "SELECT * FROM size_estimates WHERE 
keyspace_name='bad_keyspace'")
+
 @since('4.0')
 def test_set_get_concurrent_view_builders(self):
 """
diff --git a/schema_test.py b/schema_test.py
index 8aaea3d..6c9f8a1 100644
--- a/schema_test.py
+++ b/schema_test.py
@@ -4,8 +4,8 @@ import logging
 
 from cassandra.concurrent import execute_concurrent_with_args
 
-from tools.assertions import assert_invalid, assert_all, assert_one
-from dtest import Tester, create_ks
+from tools.assertions import assert_invalid, assert_all, assert_one, 
assert_none, assert_some
+from dtest import Tester, create_ks, create_cf
 
 since = pytest.mark.since
 logger = logging.getLogger(__name__)
@@ -176,6 +176,63 @@ class TestSchema(Tester):
 session.execute("USE ks")
 assert_all(session, "SELECT * FROM ts", [[1, 1, 'v1'], [1, 2, 'v2'], 
[2, 1, 'v3']])
 
+@since('3.0')
+def drop_table_reflected_in_size_estimates_test(self):
+"""
+A dropped table should result in its entries being removed from size 
estimates, on both
+nodes that are up and down at the time of the drop.
+
+@jira_ticket CASSANDRA-14905
+"""
+cluster = self.cluster
+cluster.populate(2).start()
+node1, node2 = cluster.nodelist()
+session = self.patient_exclusive_cql_connection(node1)
+create_ks(session, 'ks1', 2)
+create_ks(session, 'ks2', 2)
+create_cf(session, 'ks1.cf1', columns={'c1': 'text', 'c2': 'text'})
+create_cf(session, 'ks2.cf1', columns={'c1': 'text', 'c2': 'text'})
+create_cf(session, 'ks2.cf2', columns={'c1': 'text', 'c2': 'text'})
+
+node1.nodetool('refreshsizeestimates')
+node2.nodetool('refreshsizeestimates')
+node2.stop()
+session.execute('DROP TABLE ks2.cf1')
+session.execute('DROP KEYSPACE ks1')
+node2.start(wait_for_binary_proto=True)
+session2 = self.patient_exclusive_cql_connection(node2)
+
+session.cluster.control_connection.wait_for_schema_agreement()
+
+assert_none(session, "SELECT * FROM system.size_estimates WHERE 
keyspace_name='ks1'")
+assert_none(session, "SELECT * FROM system.size_estimates WHERE 
keyspace_name='ks2' AND table_name='cf1'")
+assert_some(session, "SELECT * FROM system.size_estimates WHERE 
keyspace_name='ks2' AND table_name='cf2'")
+assert_none(session2, "SELECT * FROM 

[jira] [Updated] (CASSANDRA-14905) If SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14905:
---
Fix Version/s: (was: 4.0.x)
   (was: 3.11.x)
   (was: 3.0.x)
   4.0
   3.11.4
   3.0.18

> If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
> size_estimates table will never be cleared for that table.
> 
>
> Key: CASSANDRA-14905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Aleksandr Sorokoumov
>Assignee: Aleksandr Sorokoumov
>Priority: Minor
> Fix For: 3.0.18, 3.11.4, 4.0
>
> Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, 
> 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, 
> 14905-4.0-testall.png
>
>
> if a node is down when a keyspace/table is dropped, it will receive the 
> schema notification before the size estimates listener is registered, so the 
> entries for the dropped keyspace/table will never be cleaned from the table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14905) If SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14905:
---
Resolution: Fixed
 Reviewers: Ariel Weisberg
Status: Resolved  (was: Ready to Commit)

Ran the dtests 
[here|https://circleci.com/gh/aweisberg/cassandra/2502#tests/containers/11] and 
[here|https://circleci.com/gh/aweisberg/cassandra/2503#tests/containers/32] as 
well as the utests.

https://circleci.com/gh/aweisberg/cassandra/2500
https://circleci.com/gh/aweisberg/cassandra/2498
https://circleci.com/gh/aweisberg/cassandra/2496

Committed as 
[ddbcff3363c5ad13bd8975e80b3f28ae8149a459|https://github.com/apache/cassandra/commit/ddbcff3363c5ad13bd8975e80b3f28ae8149a459]
 in Cassandra and 
[7a6d9002709628de2bc6af9d987a189b302e4472|https://github.com/apache/cassandra-dtest/commit/7a6d9002709628de2bc6af9d987a189b302e4472]
 in the dtests.


> If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
> size_estimates table will never be cleared for that table.
> 
>
> Key: CASSANDRA-14905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Aleksandr Sorokoumov
>Assignee: Aleksandr Sorokoumov
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, 
> 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, 
> 14905-4.0-testall.png
>
>
> if a node is down when a keyspace/table is dropped, it will receive the 
> schema notification before the size estimates listener is registered, so the 
> entries for the dropped keyspace/table will never be cleaned from the table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14905) If SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14905:
---
Status: Ready to Commit  (was: Patch Available)

> If SizeEstimatesRecorder misses a 'onDropTable' notification, the 
> size_estimates table will never be cleared for that table.
> 
>
> Key: CASSANDRA-14905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14905
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Aleksandr Sorokoumov
>Assignee: Aleksandr Sorokoumov
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, 
> 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, 
> 14905-4.0-testall.png
>
>
> if a node is down when a keyspace/table is dropped, it will receive the 
> schema notification before the size estimates listener is registered, so the 
> entries for the dropped keyspace/table will never be cleaned from the table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14981) Cassandra fires NoHostAvailable exception, but trigger receives update

[Timur Tibeyev (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742919#comment-16742919
 ] 

Timur Tibeyev commented on CASSANDRA-14981:
---

[~djoshi3]

Please check my sample repo 
[cassandra-cluster-test|https://github.com/timurt/cassandra-cluster-test]

Here are the steps;

1) Starting docker containers with `docker-compose up`

2) Connecting to any `Cassandra` node with `cqlsh`, creating keyspace with 
replication factor 2, creating table and adding trigger to table

3) Executing some simple inserts, updates, deletes to check triggers

4) Connect to `Kafka` broker using console consumer and checking record 
messages - do they correspond to `cql` queries or not

Next steps to reproduce exception

5) shut down any two nodes

6) connect to last working node with `cqlsh`

7) Execute insert queries, some of them will fail with `NoHostAvailable` 
exception, some of them not

8) Check `Kafka` consumer log, you will see all messages, even those, which 
were not written to Database

After all you can start nodes and perform `select`, data in `Cassandra` and in 
`Kafka` mismatches

Hope it will help

> Cassandra fires NoHostAvailable exception, but trigger receives update
> --
>
> Key: CASSANDRA-14981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14981
> Project: Cassandra
>  Issue Type: Bug
> Environment: cassandra 3.11.3
>Reporter: Timur Tibeyev
>Priority: Major
> Attachments: Cassandra.png, Kafka consumer.png
>
>
> Cluster
> I have cluster with three nodes, cluster is created via `docker-compose`.
> All nodes in the cluster have `UN` (up normal)  status, checked with 
> `nodetool status`.
>  
> Trigger
> Trigger receives event and produces `Kafka` record.
> All three nodes have same trigger.
>  
> Table
> Here is my CQL query, `users` table replication factor is 2
> Consistency level is one
> ```
> CREATE KEYSPACE mytestdb WITH replication = \{ 'class':'SimpleStrategy', 
> 'replication_factor':2 };
>  USE mytestdb;
> CREATE TABLE IF NOT EXISTS users (
>  id uuid,
>  username text,
>  phone text,
>  lastname text,
>  firstname text,
>  PRIMARY KEY (id)
>  );
> CREATE TRIGGER kafka_trigger ON users USING 'org.company.CassandraTrigger';
> ```
>  
> Problem
> Everything worked fine so I wanted to check edge cases, I intentionally shut 
> down 2 nodes, to check how `Cassandra` and trigger will perform.
> I started to make queries to single working node.
> Some inserts were processed correctly, but for some of them `NoHostAvailable` 
> exception occurred, but trigger received this "failed" insert event and 
> produced normal `Kafka` record
> The problem is - data was not saved to `Cassandra`, but trigger received 
> update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14982) PicklingError: Can't pickle : attribute lookup cqlshlib.copyutil.ImmutableDict failed

[Vaclav (JIRA)]

Vaclav created CASSANDRA-14982:
--

 Summary: PicklingError: Can't pickle : attribute lookup 
cqlshlib.copyutil.ImmutableDict failed
 Key: CASSANDRA-14982
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14982
 Project: Cassandra
  Issue Type: Bug
  Components: CQL/Interpreter
 Environment: cqlsh --version
cqlsh 5.0.1

 
Reporter: Vaclav
 Fix For: 3.11.3


Hello

I am trying to load records from file containing some very long lines (up to 
180_000 characters). In some cases order of lines in file causes error
'Error from server: code=2200 [Invalid query] message="Batch too large"' 
catched and printed in copyutils.py, class SendingChanel in method feed(), but 
error is just printed, records are not loaded, no error file with unimported 
rows is created and import continues. cqlsh at the end returns code 0 even not 
all rows are imported. Even that number of imported rows is wrong, it shows 
number of records in file but in fact it loaded less records. So I cannot be 
sure, based on returned code, that copy command did load all rows. My problem 
is that in this case only way to find something went wrong and data are not 
loaded correctly is to watch for error message on screen, which is problem when 
this happens in very long import script (script loading many tables).

I think when not all rows are loaded correctly return code should not be 0, or 
when records aren't loaded it should exit with error immediatelly.


$ cqlsh 127.0.0.1 --request-timeout="3600" -e "copy woc.item_container from 
'/tmp/cexport/woc/item_container.csv' with escape='\"' and null=null and 
header=True"
Reading options from the command line: \{'header': 'True', 'null': 'null', 
'escape': '"'}
Using 7 child processes

Starting copy of woc.item_container with columns [container_id, capacity, 
classes, instances, owner_id, type].
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
PicklingError: Can't pickle : 
attribute lookup cqlshlib.copyutil.ImmutableDict failed
Processed: 38420 rows; Rate: 7455 rows/s; Avg. rate: 6881 rows/s
38420 rows imported from 1 files in 5.584 seconds (0 skipped).
$ echo $?
0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org