[jira] [Commented] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047430#comment-16047430
 ] 

mck commented on CASSANDRA-11381:
-

[~jkni], work on the dtest in 
https://github.com/riptano/cassandra-dtest/pull/1479
You said you had some ideas about making it faster/more-efficient.

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[mck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-11381:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[01/10] cassandra git commit: Node running with join_ring=false and authentication can not serve requests

[mck]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 5807eca8b -> f49579df3
  refs/heads/cassandra-3.0 40ad3cf4d -> 7b9868ce2
  refs/heads/cassandra-3.11 30412b08c -> 6220108e6
  refs/heads/trunk 3e8f3c5ed -> afd68abe6


Node running with join_ring=false and authentication can not serve requests

 patch by Mick Semb Wever; reviewed by Joel Knighton for CASSANDRA-11381


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f49579df
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f49579df
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f49579df

Branch: refs/heads/cassandra-2.2
Commit: f49579df38121f88107531307437b447a7438cda
Parents: 5807eca
Author: mck 
Authored: Fri Mar 18 14:22:32 2016 +1100
Committer: mck 
Committed: Tue Jun 13 13:57:17 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d00ddcb..56890c9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.10
+ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
  * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
  * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
  * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--
diff --git a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java 
b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index 87aca21..bfd0483 100644
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@ -40,6 +40,7 @@ import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.exceptions.*;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.transport.messages.ResultMessage;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.mindrot.jbcrypt.BCrypt;
@@ -344,6 +345,9 @@ public class CassandraRoleManager implements IRoleManager
  */
 private static void setupDefaultRole()
 {
+if 
(StorageService.instance.getTokenMetadata().sortedTokens().isEmpty())
+throw new IllegalStateException("CassandraRoleManager skipped 
default role setup: no known tokens in ring");
+
 try
 {
 if (!hasExistingRoles())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 65f536b..9d2d7bb 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -191,6 +191,8 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private final StreamStateStore streamStateStore = new StreamStateStore();
 
+private final AtomicBoolean doneAuthSetup = new AtomicBoolean(false);
+
 /** This method updates the local token on disk  */
 public void setTokens(Collection tokens)
 {
@@ -657,6 +659,7 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 states.add(Pair.create(ApplicationState.STATUS, 
valueFactory.hibernate(true)));
 Gossiper.instance.addLocalApplicationStates(states);
 }
+doAuthSetup();
 logger.info("Not joining ring as requested. Use JMX 
(StorageService->joinRing()) to initiate ring joining");
 }
 }
@@ -974,12 +977,15 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private void doAuthSetup()
 {
-maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
+if (!doneAuthSetup.getAndSet(true))
+{
+maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
 
-DatabaseDescriptor.getRoleManager().setup();
-

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[mck]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6220108e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6220108e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6220108e

Branch: refs/heads/cassandra-3.11
Commit: 6220108e687e28b05d7cb0bbd254751f51a0f1d7
Parents: 30412b0 7b9868c
Author: mck 
Authored: Tue Jun 13 14:11:15 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:15:03 2017 +1000

--
 CHANGES.txt|  1 +
 .../cassandra/auth/CassandraRoleManager.java   |  4 
 .../apache/cassandra/service/StorageService.java   | 17 +++--
 3 files changed, 16 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/CHANGES.txt
--
diff --cc CHANGES.txt
index 2ffa526,26462db..f285ff0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -74,34 -40,12 +74,35 @@@ Merged from 3.0
   * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)
   * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
   * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 - * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
 + * Prevent data loss on upgrade 2.1 - 3.0 by adding component separator to 
LogRecord absolute path (CASSANDRA-13294)
 + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
 + * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
 + * Fix "multiple versions of ant detected..." when running ant test 
(CASSANDRA-13232)
 + * Coalescing strategy sleeps too much (CASSANDRA-13090)
 + * Faster StreamingHistogram (CASSANDRA-13038)
 + * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
 + * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
 + * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)
 + * Use keyspace replication settings on system.size_estimates table 
(CASSANDRA-9639)
 + * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Hint related logging should include the IP address of the destination in 
addition to
 +   host ID (CASSANDRA-13205)
 + * Reloading logback.xml does not work (CASSANDRA-13173)
 + * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)
 + * Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9 (CASSANDRA-13125)
 + * Fix UPDATE queries with empty IN restrictions (CASSANDRA-13152)
 + * Fix handling of partition with partition-level deletion plus
 +   live rows in sstabledump (CASSANDRA-13177)
 + * Provide user workaround when system_schema.columns does not contain entries
 +   for a table that's in system_schema.tables (CASSANDRA-13180)
   * Nodetool upgradesstables/scrub/compact ignores system tables 
(CASSANDRA-13410)
 - * Fix NPE issue in StorageService (CASSANDRA-13060)
 + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441)
  Merged from 2.2:
++ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
 + * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
   * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)
   * Fix weightedSize() for row-cache reported by JMX and NodeTool 
(CASSANDRA-13393)
 + * Fix JVM metric names (CASSANDRA-13103)
   * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773)
   * Discard in-flight shadow round responses (CASSANDRA-12653)
   * Don't anti-compact repaired data to avoid inconsistencies (CASSANDRA-13153)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 749c6a9,3bad040..b4878c7
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -190,12 -177,10 +190,13 @@@ public class StorageService extends Not
  private boolean isSurveyMode = 
Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false"));
  /* true if node is rebuilding and receiving data */
  private final AtomicBoolean isRebuilding = new AtomicBoolean();
 +private final AtomicBoolean isDecommissioning = new AtomicBoolean();
  
 -private 

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mck]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7b9868ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7b9868ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7b9868ce

Branch: refs/heads/trunk
Commit: 7b9868ce277d82f4ad84cb74e3179ff14d88b03e
Parents: 40ad3cf f49579d
Author: mck 
Authored: Tue Jun 13 14:04:07 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:05:12 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7b9868ce/CHANGES.txt
--
diff --cc CHANGES.txt
index 63bb33c,56890c9..26462db
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,50 -1,10 +1,51 @@@
 -2.2.10
 +3.0.14
 + * Failed unregistering mbean during drop keyspace (CASSANDRA-13346)
 + * nodetool scrub/cleanup/upgradesstables exit code is wrong (CASSANDRA-13542)
 + * Fix the reported number of sstable data files accessed per read 
(CASSANDRA-13120)
 + * Fix schema digest mismatch during rolling upgrades from versions before 
3.0.12 (CASSANDRA-13559)
 + * Upgrade JNA version to 4.4.0 (CASSANDRA-13072)
 + * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
 + * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
 + * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
 + * Add storage port options to sstableloader (CASSANDRA-13518)
 + * Properly handle quoted index names in cqlsh DESCRIBE output 
(CASSANDRA-12847)
 + * Avoid reading static row twice from old format sstables (CASSANDRA-13236)
 + * Fix NPE in StorageService.excise() (CASSANDRA-13163)
 + * Expire OutboundTcpConnection messages by a single Thread (CASSANDRA-13265)
 + * Fail repair if insufficient responses received (CASSANDRA-13397)
 + * Fix SSTableLoader fail when the loaded table contains dropped columns 
(CASSANDRA-13276)
 + * Avoid name clashes in CassandraIndexTest (CASSANDRA-13427)
 + * Handling partially written hint files (CASSANDRA-12728)
 + * Interrupt replaying hints on decommission (CASSANDRA-13308)
 + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441)
 +Merged from 2.2:
+  * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
   * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
 - * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
 +
 +3.0.13
 + * Make reading of range tombstones more reliable (CASSANDRA-12811)
 + * Fix startup problems due to schema tables not completely flushed 
(CASSANDRA-12213)
 + * Fix view builder bug that can filter out data on restart (CASSANDRA-13405)
 + * Fix 2i page size calculation when there are no regular columns 
(CASSANDRA-13400)
 + * Fix the conversion of 2.X expired rows without regular column data 
(CASSANDRA-13395)
 + * Fix hint delivery when using ext+internal IPs with prefer_local enabled 
(CASSANDRA-13020)
 + * Fix possible NPE on upgrade to 3.0/3.X in case of IO errors 
(CASSANDRA-13389)
 + * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
 + * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
 + * Fix code to not exchange schema across major versions (CASSANDRA-13274)
 + * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
 + * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
 + * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
 + * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)
 + * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)
 + * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
 + * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
 + * Nodetool upgradesstables/scrub/compact ignores system tables 
(CASSANDRA-13410)
 + * Fix NPE issue in StorageService (CASSANDRA-13060)
 +Merged from 2.2:
   * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)
   * Fix weightedSize() for row-cache reported by JMX and NodeTool 
(CASSANDRA-13393)
 - * Fix JVM metric paths (CASSANDRA-13103)
   * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773)
   * Discard in-flight shadow round responses 

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[mck]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6220108e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6220108e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6220108e

Branch: refs/heads/trunk
Commit: 6220108e687e28b05d7cb0bbd254751f51a0f1d7
Parents: 30412b0 7b9868c
Author: mck 
Authored: Tue Jun 13 14:11:15 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:15:03 2017 +1000

--
 CHANGES.txt|  1 +
 .../cassandra/auth/CassandraRoleManager.java   |  4 
 .../apache/cassandra/service/StorageService.java   | 17 +++--
 3 files changed, 16 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/CHANGES.txt
--
diff --cc CHANGES.txt
index 2ffa526,26462db..f285ff0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -74,34 -40,12 +74,35 @@@ Merged from 3.0
   * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)
   * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
   * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 - * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
 + * Prevent data loss on upgrade 2.1 - 3.0 by adding component separator to 
LogRecord absolute path (CASSANDRA-13294)
 + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
 + * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
 + * Fix "multiple versions of ant detected..." when running ant test 
(CASSANDRA-13232)
 + * Coalescing strategy sleeps too much (CASSANDRA-13090)
 + * Faster StreamingHistogram (CASSANDRA-13038)
 + * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
 + * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
 + * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)
 + * Use keyspace replication settings on system.size_estimates table 
(CASSANDRA-9639)
 + * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Hint related logging should include the IP address of the destination in 
addition to
 +   host ID (CASSANDRA-13205)
 + * Reloading logback.xml does not work (CASSANDRA-13173)
 + * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)
 + * Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9 (CASSANDRA-13125)
 + * Fix UPDATE queries with empty IN restrictions (CASSANDRA-13152)
 + * Fix handling of partition with partition-level deletion plus
 +   live rows in sstabledump (CASSANDRA-13177)
 + * Provide user workaround when system_schema.columns does not contain entries
 +   for a table that's in system_schema.tables (CASSANDRA-13180)
   * Nodetool upgradesstables/scrub/compact ignores system tables 
(CASSANDRA-13410)
 - * Fix NPE issue in StorageService (CASSANDRA-13060)
 + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441)
  Merged from 2.2:
++ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
 + * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
   * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)
   * Fix weightedSize() for row-cache reported by JMX and NodeTool 
(CASSANDRA-13393)
 + * Fix JVM metric names (CASSANDRA-13103)
   * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773)
   * Discard in-flight shadow round responses (CASSANDRA-12653)
   * Don't anti-compact repaired data to avoid inconsistencies (CASSANDRA-13153)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6220108e/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 749c6a9,3bad040..b4878c7
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -190,12 -177,10 +190,13 @@@ public class StorageService extends Not
  private boolean isSurveyMode = 
Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false"));
  /* true if node is rebuilding and receiving data */
  private final AtomicBoolean isRebuilding = new AtomicBoolean();
 +private final AtomicBoolean isDecommissioning = new AtomicBoolean();
  
 -private boolean 

[04/10] cassandra git commit: Node running with join_ring=false and authentication can not serve requests

[mck]

Node running with join_ring=false and authentication can not serve requests

 patch by Mick Semb Wever; reviewed by Joel Knighton for CASSANDRA-11381


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f49579df
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f49579df
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f49579df

Branch: refs/heads/trunk
Commit: f49579df38121f88107531307437b447a7438cda
Parents: 5807eca
Author: mck 
Authored: Fri Mar 18 14:22:32 2016 +1100
Committer: mck 
Committed: Tue Jun 13 13:57:17 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d00ddcb..56890c9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.10
+ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
  * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
  * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
  * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--
diff --git a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java 
b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index 87aca21..bfd0483 100644
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@ -40,6 +40,7 @@ import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.exceptions.*;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.transport.messages.ResultMessage;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.mindrot.jbcrypt.BCrypt;
@@ -344,6 +345,9 @@ public class CassandraRoleManager implements IRoleManager
  */
 private static void setupDefaultRole()
 {
+if 
(StorageService.instance.getTokenMetadata().sortedTokens().isEmpty())
+throw new IllegalStateException("CassandraRoleManager skipped 
default role setup: no known tokens in ring");
+
 try
 {
 if (!hasExistingRoles())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 65f536b..9d2d7bb 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -191,6 +191,8 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private final StreamStateStore streamStateStore = new StreamStateStore();
 
+private final AtomicBoolean doneAuthSetup = new AtomicBoolean(false);
+
 /** This method updates the local token on disk  */
 public void setTokens(Collection tokens)
 {
@@ -657,6 +659,7 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 states.add(Pair.create(ApplicationState.STATUS, 
valueFactory.hibernate(true)));
 Gossiper.instance.addLocalApplicationStates(states);
 }
+doAuthSetup();
 logger.info("Not joining ring as requested. Use JMX 
(StorageService->joinRing()) to initiate ring joining");
 }
 }
@@ -974,12 +977,15 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private void doAuthSetup()
 {
-maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
+if (!doneAuthSetup.getAndSet(true))
+{
+maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
 
-DatabaseDescriptor.getRoleManager().setup();
-DatabaseDescriptor.getAuthenticator().setup();
-DatabaseDescriptor.getAuthorizer().setup();
-MigrationManager.instance.register(new AuthMigrationListener());
+DatabaseDescriptor.getRoleManager().setup();
+

[03/10] cassandra git commit: Node running with join_ring=false and authentication can not serve requests

[mck]

Node running with join_ring=false and authentication can not serve requests

 patch by Mick Semb Wever; reviewed by Joel Knighton for CASSANDRA-11381


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f49579df
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f49579df
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f49579df

Branch: refs/heads/cassandra-3.11
Commit: f49579df38121f88107531307437b447a7438cda
Parents: 5807eca
Author: mck 
Authored: Fri Mar 18 14:22:32 2016 +1100
Committer: mck 
Committed: Tue Jun 13 13:57:17 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d00ddcb..56890c9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.10
+ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
  * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
  * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
  * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--
diff --git a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java 
b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index 87aca21..bfd0483 100644
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@ -40,6 +40,7 @@ import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.exceptions.*;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.transport.messages.ResultMessage;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.mindrot.jbcrypt.BCrypt;
@@ -344,6 +345,9 @@ public class CassandraRoleManager implements IRoleManager
  */
 private static void setupDefaultRole()
 {
+if 
(StorageService.instance.getTokenMetadata().sortedTokens().isEmpty())
+throw new IllegalStateException("CassandraRoleManager skipped 
default role setup: no known tokens in ring");
+
 try
 {
 if (!hasExistingRoles())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 65f536b..9d2d7bb 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -191,6 +191,8 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private final StreamStateStore streamStateStore = new StreamStateStore();
 
+private final AtomicBoolean doneAuthSetup = new AtomicBoolean(false);
+
 /** This method updates the local token on disk  */
 public void setTokens(Collection tokens)
 {
@@ -657,6 +659,7 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 states.add(Pair.create(ApplicationState.STATUS, 
valueFactory.hibernate(true)));
 Gossiper.instance.addLocalApplicationStates(states);
 }
+doAuthSetup();
 logger.info("Not joining ring as requested. Use JMX 
(StorageService->joinRing()) to initiate ring joining");
 }
 }
@@ -974,12 +977,15 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private void doAuthSetup()
 {
-maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
+if (!doneAuthSetup.getAndSet(true))
+{
+maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
 
-DatabaseDescriptor.getRoleManager().setup();
-DatabaseDescriptor.getAuthenticator().setup();
-DatabaseDescriptor.getAuthorizer().setup();
-MigrationManager.instance.register(new AuthMigrationListener());
+DatabaseDescriptor.getRoleManager().setup();
+

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mck]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7b9868ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7b9868ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7b9868ce

Branch: refs/heads/cassandra-3.11
Commit: 7b9868ce277d82f4ad84cb74e3179ff14d88b03e
Parents: 40ad3cf f49579d
Author: mck 
Authored: Tue Jun 13 14:04:07 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:05:12 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7b9868ce/CHANGES.txt
--
diff --cc CHANGES.txt
index 63bb33c,56890c9..26462db
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,50 -1,10 +1,51 @@@
 -2.2.10
 +3.0.14
 + * Failed unregistering mbean during drop keyspace (CASSANDRA-13346)
 + * nodetool scrub/cleanup/upgradesstables exit code is wrong (CASSANDRA-13542)
 + * Fix the reported number of sstable data files accessed per read 
(CASSANDRA-13120)
 + * Fix schema digest mismatch during rolling upgrades from versions before 
3.0.12 (CASSANDRA-13559)
 + * Upgrade JNA version to 4.4.0 (CASSANDRA-13072)
 + * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
 + * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
 + * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
 + * Add storage port options to sstableloader (CASSANDRA-13518)
 + * Properly handle quoted index names in cqlsh DESCRIBE output 
(CASSANDRA-12847)
 + * Avoid reading static row twice from old format sstables (CASSANDRA-13236)
 + * Fix NPE in StorageService.excise() (CASSANDRA-13163)
 + * Expire OutboundTcpConnection messages by a single Thread (CASSANDRA-13265)
 + * Fail repair if insufficient responses received (CASSANDRA-13397)
 + * Fix SSTableLoader fail when the loaded table contains dropped columns 
(CASSANDRA-13276)
 + * Avoid name clashes in CassandraIndexTest (CASSANDRA-13427)
 + * Handling partially written hint files (CASSANDRA-12728)
 + * Interrupt replaying hints on decommission (CASSANDRA-13308)
 + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441)
 +Merged from 2.2:
+  * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
   * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
 - * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
 +
 +3.0.13
 + * Make reading of range tombstones more reliable (CASSANDRA-12811)
 + * Fix startup problems due to schema tables not completely flushed 
(CASSANDRA-12213)
 + * Fix view builder bug that can filter out data on restart (CASSANDRA-13405)
 + * Fix 2i page size calculation when there are no regular columns 
(CASSANDRA-13400)
 + * Fix the conversion of 2.X expired rows without regular column data 
(CASSANDRA-13395)
 + * Fix hint delivery when using ext+internal IPs with prefer_local enabled 
(CASSANDRA-13020)
 + * Fix possible NPE on upgrade to 3.0/3.X in case of IO errors 
(CASSANDRA-13389)
 + * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
 + * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
 + * Fix code to not exchange schema across major versions (CASSANDRA-13274)
 + * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
 + * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
 + * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
 + * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)
 + * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)
 + * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
 + * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
 + * Nodetool upgradesstables/scrub/compact ignores system tables 
(CASSANDRA-13410)
 + * Fix NPE issue in StorageService (CASSANDRA-13060)
 +Merged from 2.2:
   * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)
   * Fix weightedSize() for row-cache reported by JMX and NodeTool 
(CASSANDRA-13393)
 - * Fix JVM metric paths (CASSANDRA-13103)
   * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773)
   * Discard in-flight shadow round responses 

[02/10] cassandra git commit: Node running with join_ring=false and authentication can not serve requests

[mck]

Node running with join_ring=false and authentication can not serve requests

 patch by Mick Semb Wever; reviewed by Joel Knighton for CASSANDRA-11381


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f49579df
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f49579df
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f49579df

Branch: refs/heads/cassandra-3.0
Commit: f49579df38121f88107531307437b447a7438cda
Parents: 5807eca
Author: mck 
Authored: Fri Mar 18 14:22:32 2016 +1100
Committer: mck 
Committed: Tue Jun 13 13:57:17 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d00ddcb..56890c9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.10
+ * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
  * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
  * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
  * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--
diff --git a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java 
b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index 87aca21..bfd0483 100644
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@ -40,6 +40,7 @@ import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.exceptions.*;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.transport.messages.ResultMessage;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.mindrot.jbcrypt.BCrypt;
@@ -344,6 +345,9 @@ public class CassandraRoleManager implements IRoleManager
  */
 private static void setupDefaultRole()
 {
+if 
(StorageService.instance.getTokenMetadata().sortedTokens().isEmpty())
+throw new IllegalStateException("CassandraRoleManager skipped 
default role setup: no known tokens in ring");
+
 try
 {
 if (!hasExistingRoles())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f49579df/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 65f536b..9d2d7bb 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -191,6 +191,8 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private final StreamStateStore streamStateStore = new StreamStateStore();
 
+private final AtomicBoolean doneAuthSetup = new AtomicBoolean(false);
+
 /** This method updates the local token on disk  */
 public void setTokens(Collection tokens)
 {
@@ -657,6 +659,7 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 states.add(Pair.create(ApplicationState.STATUS, 
valueFactory.hibernate(true)));
 Gossiper.instance.addLocalApplicationStates(states);
 }
+doAuthSetup();
 logger.info("Not joining ring as requested. Use JMX 
(StorageService->joinRing()) to initiate ring joining");
 }
 }
@@ -974,12 +977,15 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 private void doAuthSetup()
 {
-maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
+if (!doneAuthSetup.getAndSet(true))
+{
+maybeAddOrUpdateKeyspace(AuthKeyspace.definition());
 
-DatabaseDescriptor.getRoleManager().setup();
-DatabaseDescriptor.getAuthenticator().setup();
-DatabaseDescriptor.getAuthorizer().setup();
-MigrationManager.instance.register(new AuthMigrationListener());
+DatabaseDescriptor.getRoleManager().setup();
+

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[mck]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/afd68abe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/afd68abe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/afd68abe

Branch: refs/heads/trunk
Commit: afd68abe60742c6deb6357ba4605268dfb3d06ea
Parents: 3e8f3c5 6220108
Author: mck 
Authored: Tue Jun 13 14:15:57 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:19:58 2017 +1000

--
 CHANGES.txt|  1 +
 .../cassandra/auth/CassandraRoleManager.java   |  4 
 .../apache/cassandra/service/StorageService.java   | 17 +++--
 3 files changed, 16 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/afd68abe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/afd68abe/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
--
diff --cc src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index 0591760,777ed05..433290e
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@@ -39,7 -39,9 +39,8 @@@ import org.apache.cassandra.cql3.statem
  import org.apache.cassandra.db.ConsistencyLevel;
  import org.apache.cassandra.db.marshal.UTF8Type;
  import org.apache.cassandra.exceptions.*;
 -import org.apache.cassandra.net.MessagingService;
  import org.apache.cassandra.service.QueryState;
+ import org.apache.cassandra.service.StorageService;
  import org.apache.cassandra.transport.messages.ResultMessage;
  import org.apache.cassandra.utils.ByteBufferUtil;
  import org.mindrot.jbcrypt.BCrypt;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/afd68abe/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 8914440,b4878c7..a7d4e9d
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -1032,13 -1083,16 +1034,16 @@@ public class StorageService extends Not
  
  private void doAuthSetup()
  {
- maybeAddOrUpdateKeyspace(AuthKeyspace.metadata());
+ if (!authSetupCalled.getAndSet(true))
+ {
+ maybeAddOrUpdateKeyspace(AuthKeyspace.metadata());
  
- DatabaseDescriptor.getRoleManager().setup();
- DatabaseDescriptor.getAuthenticator().setup();
- DatabaseDescriptor.getAuthorizer().setup();
- Schema.instance.registerListener(new AuthSchemaChangeListener());
- authSetupComplete = true;
+ DatabaseDescriptor.getRoleManager().setup();
+ DatabaseDescriptor.getAuthenticator().setup();
+ DatabaseDescriptor.getAuthorizer().setup();
 -MigrationManager.instance.register(new AuthMigrationListener());
++Schema.instance.registerListener(new AuthSchemaChangeListener());
+ authSetupComplete = true;
+ }
  }
  
  public boolean isAuthSetupComplete()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mck]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7b9868ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7b9868ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7b9868ce

Branch: refs/heads/cassandra-3.0
Commit: 7b9868ce277d82f4ad84cb74e3179ff14d88b03e
Parents: 40ad3cf f49579d
Author: mck 
Authored: Tue Jun 13 14:04:07 2017 +1000
Committer: mck 
Committed: Tue Jun 13 14:05:12 2017 +1000

--
 CHANGES.txt |  1 +
 .../apache/cassandra/auth/CassandraRoleManager.java |  4 
 .../apache/cassandra/service/StorageService.java| 16 +++-
 3 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7b9868ce/CHANGES.txt
--
diff --cc CHANGES.txt
index 63bb33c,56890c9..26462db
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,50 -1,10 +1,51 @@@
 -2.2.10
 +3.0.14
 + * Failed unregistering mbean during drop keyspace (CASSANDRA-13346)
 + * nodetool scrub/cleanup/upgradesstables exit code is wrong (CASSANDRA-13542)
 + * Fix the reported number of sstable data files accessed per read 
(CASSANDRA-13120)
 + * Fix schema digest mismatch during rolling upgrades from versions before 
3.0.12 (CASSANDRA-13559)
 + * Upgrade JNA version to 4.4.0 (CASSANDRA-13072)
 + * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
 + * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
 + * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
 + * Add storage port options to sstableloader (CASSANDRA-13518)
 + * Properly handle quoted index names in cqlsh DESCRIBE output 
(CASSANDRA-12847)
 + * Avoid reading static row twice from old format sstables (CASSANDRA-13236)
 + * Fix NPE in StorageService.excise() (CASSANDRA-13163)
 + * Expire OutboundTcpConnection messages by a single Thread (CASSANDRA-13265)
 + * Fail repair if insufficient responses received (CASSANDRA-13397)
 + * Fix SSTableLoader fail when the loaded table contains dropped columns 
(CASSANDRA-13276)
 + * Avoid name clashes in CassandraIndexTest (CASSANDRA-13427)
 + * Handling partially written hint files (CASSANDRA-12728)
 + * Interrupt replaying hints on decommission (CASSANDRA-13308)
 + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441)
 +Merged from 2.2:
+  * Nodes started with join_ring=False should be able to serve requests when 
authentication is enabled (CASSANDRA-11381)
   * cqlsh COPY FROM: increment error count only for failures, not for attempts 
(CASSANDRA-13209)
 - * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119)
 +
 +3.0.13
 + * Make reading of range tombstones more reliable (CASSANDRA-12811)
 + * Fix startup problems due to schema tables not completely flushed 
(CASSANDRA-12213)
 + * Fix view builder bug that can filter out data on restart (CASSANDRA-13405)
 + * Fix 2i page size calculation when there are no regular columns 
(CASSANDRA-13400)
 + * Fix the conversion of 2.X expired rows without regular column data 
(CASSANDRA-13395)
 + * Fix hint delivery when using ext+internal IPs with prefer_local enabled 
(CASSANDRA-13020)
 + * Fix possible NPE on upgrade to 3.0/3.X in case of IO errors 
(CASSANDRA-13389)
 + * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
 + * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
 + * Fix code to not exchange schema across major versions (CASSANDRA-13274)
 + * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
 + * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
 + * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
 + * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)
 + * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)
 + * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
 + * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
 + * Nodetool upgradesstables/scrub/compact ignores system tables 
(CASSANDRA-13410)
 + * Fix NPE issue in StorageService (CASSANDRA-13060)
 +Merged from 2.2:
   * Avoid starting gossiper in RemoveTest (CASSANDRA-13407)
   * Fix weightedSize() for row-cache reported by JMX and NodeTool 
(CASSANDRA-13393)
 - * Fix JVM metric paths (CASSANDRA-13103)
   * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773)
   * Discard in-flight shadow round responses 

[jira] [Commented] (CASSANDRA-13068) Fully expired sstable not dropped when running out of disk space

[Krishna Dattu Koneru (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047380#comment-16047380
 ] 

Krishna Dattu Koneru commented on CASSANDRA-13068:
--

Hi [~krummas] , I work with [~Lerh Low] . He wanted me to continue working on 
this.  I have uploaded patches on github for  [trunk| 
https://github.com/apache/cassandra/compare/trunk...krishna-koneru:cassandra-trunk-13068
 ] and [3.11 | 
https://github.com/apache/cassandra/compare/cassandra-3.11...krishna-koneru:cassandra-3.11-13068]
  branches .



I had to make few more changes to what was initially implemented by [~Lerh Low] 
to address your comments.

- renamed {{CompactionTask#checkAvailableDiskSpace()}} to 
{{CompactionTask#buildCompactionCandidatesForAvailableDiskSpace()}} as the 
method checks and might also modify the list of SSTables that will be compacted.

- {{buildCompactionCandidatesForAvailableDiskSpace()}} does not throw exception 
 if there is no diskspace to compact non-expired SSTables when there are fully 
expired SSTables.This means, instead of throwing exception, we try to compact 
fully expired SSTables which might free up some space to allow further 
compactions. 

> Fully expired sstable not dropped when running out of disk space
> 
>
> Key: CASSANDRA-13068
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13068
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Marcus Eriksson
>Assignee: Lerh Chuan Low
>  Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> If a fully expired sstable is larger than the remaining disk space we won't 
> run the compaction that can drop the sstable (ie, in our disk space check 
> should not include the fully expired sstables)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047363#comment-16047363
 ] 

mck commented on CASSANDRA-11381:
-

{quote}one minor nit that can be fixed on commit: new CHANGES.txt entries 
should go at the top of the list.{quote}

I had no idea! :-) One of those tribal habits that one either notices or 
doesn't… thanks for pointing it out.

Will correct CHANGES.txt, and commit…

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12148) Improve determinism of CDC data availability

[Jay Zhuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047231#comment-16047231
 ] 

Jay Zhuang commented on CASSANDRA-12148:


[~JoshuaMcKenzie] thanks. It would be great to have this feature in 4.0.

> Improve determinism of CDC data availability
> 
>
> Key: CASSANDRA-12148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12148
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Joshua McKenzie
>Assignee: Joshua McKenzie
> Fix For: 4.x
>
>
> The latency with which CDC data becomes available has a known limitation due 
> to our reliance on CommitLogSegments being discarded to have the data 
> available in cdc_raw: if a slowly written table co-habitates a 
> CommitLogSegment with CDC data, the CommitLogSegment won't be flushed until 
> we hit either memory pressure on memtables or CommitLog limit pressure. 
> Ultimately, this leaves a non-deterministic element to when data becomes 
> available for CDC consumption unless a consumer parses live CommitLogSegments.
> To work around this limitation and make semi-realtime CDC consumption more 
> friendly to end-users, I propose we extend CDC as follows:
> h6. High level:
> * Consumers parse hard links of active CommitLogSegments in cdc_raw instead 
> of waiting for flush/discard and file move
> * C* stores an offset of the highest seen CDC mutation in a separate idx file 
> per commit log segment in cdc_raw. Clients tail this index file, delta their 
> local last parsed offset on change, and parse the corresponding commit log 
> segment using their last parsed offset as min
> * C* flags that index file with an offset and DONE when the file is flushed 
> so clients know when they can clean up
> h6. Details:
> * On creation of a CommitLogSegment, also hard-link the file in cdc_raw
> * On first write of a CDC-enabled mutation to a segment, we:
> ** Flag it as {{CDCState.CONTAINS}}
> ** Set a long tracking the {{CommitLogPosition}} of the 1st CDC-enabled 
> mutation in the log
> ** Set a long in the CommitLogSegment tracking the offset of the end of the 
> last written CDC mutation in the segment if higher than the previously known 
> highest CDC offset
> * On subsequent writes to the segment, we update the offset of the highest 
> known CDC data
> * On CommitLogSegment fsync, we write a file in cdc_raw as 
> _cdc.idx containing the min offset and end offset fsynced to 
> disk per file
> * On segment discard, if CDCState == {{CDCState.PERMITTED}}, delete both the 
> segment in commitlog and in cdc_raw
> * On segment discard, if CDCState == {{CDCState.CONTAINS}}, delete the 
> segment in commitlog and update the _cdc.idx file w/end offset 
> and a DONE marker
> * On segment replay, store the highest end offset of seen CDC-enabled 
> mutations from a segment and write that to _cdc.idx on 
> completion of segment replay. This should bridge the potential correctness 
> gap of a node writing to a segment and then dying before it can write the 
> _cdc.idx file.
> This should allow clients to skip the beginning of a file to the 1st CDC 
> mutation, track an offset of how far they've parsed, delta against the 
> _cdc.idx file end offset, and use that as a determinant on when to parse new 
> CDC data. Any existing clients written to the initial implementation of CDC 
> need only add the _cdc.idx logic and checking for DONE marker 
> to their code, so the burden on users to update to support this should be 
> quite small for the benefit of having data available as soon as it's fsynced 
> instead of at a non-deterministic time when potentially unrelated tables are 
> flushed.
> Finally, we should look into extending the interface on CommitLogReader to be 
> more friendly for realtime parsing, perhaps supporting taking a 
> CommitLogDescriptor and RandomAccessReader and resuming readSection calls, 
> assuming the reader is at the start of a SyncSegment. Would probably also 
> need to rewind to the start of the segment before returning so subsequent 
> calls would respect this contract. This would skip needing to deserialize the 
> descriptor and all completed SyncSegments to get to the root of the desired 
> segment for parsing.
> One alternative we discussed offline - instead of just storing the highest 
> seen CDC offset, we could instead store an offset per CDC mutation 
> (potentially delta encoded) in the idx file to allow clients to seek and only 
> parse the mutations with CDC enabled. My hunch is that the performance delta 
> from doing so wouldn't justify the complexity given the SyncSegment 
> deserialization and seeking restrictions in the compressed and encrypted 
> cases as mentioned above.
> The only complication I can think of with the above design is uncompressed 
> mmapped CommitLogSegments on Windows 

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restarts

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Summary: Deadlock during CommitLog replay when Cassandra restarts  (was: 
Deadlock during CommitLog replay when Cassandra restart)

> Deadlock during CommitLog replay when Cassandra restarts
> 
>
> Key: CASSANDRA-13587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
> Fix For: 3.0.14
>
> Attachments: 13587-3.0.txt, Reproduce_CASSANDRA-13587.txt
>
>
> Possible deadlock found when Cassandra is replaying commit log and at the 
> same time Mutation gets triggered by 
> SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
> restart hangs forever
> Please find details of stack trace here:
> *Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
> and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
> but there are no Commitlog Segments available because {{createReserveSegments 
> (CommitLogSegmentManager.java)}} is not yet {{true}} 
> Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
> please note this thread has already started {{writeOrder}}
> {quote}
> "pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
> waiting on condition [0x7fe1389a]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at 
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
> at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at 
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> *Frame#2* This thread is trying to recover commit logs and as a result it 
> tries to flush Memtable by calling following code:
> {{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
> As a result Frame#3 (below) gets created
> {quote}
> "main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
> condition [0x7fe1c94a1000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for  <0x0006370da0c0> (a 
> com.google.common.util.concurrent.ListenableFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
>

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Description: 
Possible deadlock found when Cassandra is replaying commit log and at the same 
time Mutation gets triggered by 
SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
restart hangs forever

Please find details of stack trace here:

*Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
but there are no Commitlog Segments available because {{createReserveSegments 
(CommitLogSegmentManager.java)}} is not yet {{true}} 

Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
please note this thread has already started {{writeOrder}}

{quote}
"pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
waiting on condition [0x7fe1389a]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at 
org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
at 
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
at 
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
at 
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
at 
org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
at 
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{quote}

*Frame#2* This thread is trying to recover commit logs and as a result it tries 
to flush Memtable by calling following code:
{{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
As a result Frame#3 (below) gets created

{quote}
"main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
condition [0x7fe1c94a1000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for  <0x0006370da0c0> (a 
com.google.common.util.concurrent.ListenableFutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:182)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:295)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
{quote}

*Frame#3* This thread is waiting at {{writeBarrier.await();}} in 
{{ColumnFamilyStore.java:1027}} 
but {{writeBarrier}} is locked by thread in Frame#1, and Frame#1 thread is 
waiting for more CommitlogSegements to be available. 
Frame#1 thread will not get new segment because variable 

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Attachment: Reproduce_CASSANDRA-13587.txt

Reproducible steps patch

> Deadlock during CommitLog replay when Cassandra restart
> ---
>
> Key: CASSANDRA-13587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
> Fix For: 3.0.14
>
> Attachments: 13587-3.0.txt, Reproduce_CASSANDRA-13587.txt
>
>
> Possible deadlock found when Cassandra is replaying commit log and at the 
> same time Mutation gets triggered by 
> SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
> restart hangs forever
> Please find details of stack trace here:
> *Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
> and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
> but there are no Commitlog Segments available because {{createReserveSegments 
> (CommitLogSegmentManager.java)}} is not yet {{true}} 
> Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
> please note this thread has already started {{writeOrder}}
> {quote}
> "pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
> waiting on condition [0x7fe1389a]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at 
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
> at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at 
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> *Frame#2* This thread is trying to recover commit logs and as a result it 
> tries to flush Memtable by calling following code:
> {{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
> As a result Frame#3 (below) gets created
> {quote}
> "main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
> condition [0x7fe1c94a1000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for  <0x0006370da0c0> (a 
> com.google.common.util.concurrent.ListenableFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
> at 
> 

[jira] [Issue Comment Deleted] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Comment: was deleted

(was: Reproducible steps patch)

> Deadlock during CommitLog replay when Cassandra restart
> ---
>
> Key: CASSANDRA-13587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
> Fix For: 3.0.14
>
> Attachments: 13587-3.0.txt, Reproduce_CASSANDRA-13587.txt
>
>
> Possible deadlock found when Cassandra is replaying commit log and at the 
> same time Mutation gets triggered by 
> SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
> restart hangs forever
> Please find details of stack trace here:
> *Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
> and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
> but there are no Commitlog Segments available because {{createReserveSegments 
> (CommitLogSegmentManager.java)}} is not yet {{true}} 
> Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
> please note this thread has already started {{writeOrder}}
> {quote}
> "pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
> waiting on condition [0x7fe1389a]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at 
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
> at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at 
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> *Frame#2* This thread is trying to recover commit logs and as a result it 
> tries to flush Memtable by calling following code:
> {{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
> As a result Frame#3 (below) gets created
> {quote}
> "main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
> condition [0x7fe1c94a1000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for  <0x0006370da0c0> (a 
> com.google.common.util.concurrent.ListenableFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
> at 
> 

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Attachment: (was: Reproduce_CASSANDRA-13587.txt)

> Deadlock during CommitLog replay when Cassandra restart
> ---
>
> Key: CASSANDRA-13587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
> Fix For: 3.0.14
>
> Attachments: 13587-3.0.txt, Reproduce_CASSANDRA-13587.txt
>
>
> Possible deadlock found when Cassandra is replaying commit log and at the 
> same time Mutation gets triggered by 
> SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
> restart hangs forever
> Please find details of stack trace here:
> *Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
> and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
> but there are no Commitlog Segments available because {{createReserveSegments 
> (CommitLogSegmentManager.java)}} is not yet {{true}} 
> Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
> please note this thread has already started {{writeOrder}}
> {quote}
> "pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
> waiting on condition [0x7fe1389a]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at 
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
> at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at 
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> *Frame#2* This thread is trying to recover commit logs and as a result it 
> tries to flush Memtable by calling following code:
> {{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
> As a result Frame#3 (below) gets created
> {quote}
> "main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
> condition [0x7fe1c94a1000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for  <0x0006370da0c0> (a 
> com.google.common.util.concurrent.ListenableFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
> at 
> 

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Attachment: Reproduce_CASSANDRA-13587.txt

Reproducible steps patch

> Deadlock during CommitLog replay when Cassandra restart
> ---
>
> Key: CASSANDRA-13587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
> Fix For: 3.0.14
>
> Attachments: 13587-3.0.txt, Reproduce_CASSANDRA-13587.txt
>
>
> Possible deadlock found when Cassandra is replaying commit log and at the 
> same time Mutation gets triggered by 
> SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
> restart hangs forever
> Please find details of stack trace here:
> *Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
> and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
> but there are no Commitlog Segments available because {{createReserveSegments 
> (CommitLogSegmentManager.java)}} is not yet {{true}} 
> Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
> please note this thread has already started {{writeOrder}}
> {quote}
> "pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
> waiting on condition [0x7fe1389a]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at 
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
> at 
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
> at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at 
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> *Frame#2* This thread is trying to recover commit logs and as a result it 
> tries to flush Memtable by calling following code:
> {{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
> As a result Frame#3 (below) gets created
> {quote}
> "main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
> condition [0x7fe1c94a1000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for  <0x0006370da0c0> (a 
> com.google.common.util.concurrent.ListenableFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
> at 
> 

[jira] [Updated] (CASSANDRA-13587) Deadlock during CommitLog replay when Cassandra restart

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13587:
--
Description: 
Possible deadlock found when Cassandra is replaying commit log and at the same 
time Mutation gets triggered by 
SSTableReader(SystemKeyspace.persistSSTableReadMeter). As a result Cassandra 
restart hangs forever

Please find details of stack trace here:

*Frame#1* This thread is trying to apply {{persistSSTableReadMeter}} mutation 
and as a result it has called {{writeOrder.start()}} in {{Keyspace.java:533}}
but there are no Commitlog Segments available because {{createReserveSegments 
(CommitLogSegmentManager.java)}} is not yet {{true}} 

Hence this thread is blocked on {{createReserveSegments}} to become {{true}}, 
please note this thread has already started {{writeOrder}}

{quote}
"pool-11-thread-1" #251 prio=5 os_prio=0 tid=0x7fe128478400 nid=0x1b274 
waiting on condition [0x7fe1389a]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at 
org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
at 
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.advanceAllocatingFrom(CommitLogSegmentManager.java:277)
at 
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.allocate(CommitLogSegmentManager.java:196)
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:260)
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:540)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
at 
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
at 
org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1181)
at 
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2202)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{quote}

*Frame#2* This thread is trying to recover commit logs and as a result it tries 
to flush Memtable by calling following code:
{{futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());}}
As a result Frame#3 (below) gets created

{quote}
"main" #1 prio=5 os_prio=0 tid=0x7fe1c64ec400 nid=0x1af29 waiting on 
condition [0x7fe1c94a1000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for  <0x0006370da0c0> (a 
com.google.common.util.concurrent.ListenableFutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:383)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:207)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:182)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:295)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
{quote}

*Frame#3* This thread is waiting at {{writeBarrier.await();}} in 
{{ColumnFamilyStore.java:1027}} 
but {{writeBarrier}} is locked by thread in Frame#1, and Frame#1 thread is 
waiting for more CommitlogSegements to be available. 
Frame#1 thread will not get new segment because variable 

[jira] [Updated] (CASSANDRA-12148) Improve determinism of CDC data availability

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-12148:

Fix Version/s: 4.x

> Improve determinism of CDC data availability
> 
>
> Key: CASSANDRA-12148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12148
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Joshua McKenzie
>Assignee: Joshua McKenzie
> Fix For: 4.x
>
>
> The latency with which CDC data becomes available has a known limitation due 
> to our reliance on CommitLogSegments being discarded to have the data 
> available in cdc_raw: if a slowly written table co-habitates a 
> CommitLogSegment with CDC data, the CommitLogSegment won't be flushed until 
> we hit either memory pressure on memtables or CommitLog limit pressure. 
> Ultimately, this leaves a non-deterministic element to when data becomes 
> available for CDC consumption unless a consumer parses live CommitLogSegments.
> To work around this limitation and make semi-realtime CDC consumption more 
> friendly to end-users, I propose we extend CDC as follows:
> h6. High level:
> * Consumers parse hard links of active CommitLogSegments in cdc_raw instead 
> of waiting for flush/discard and file move
> * C* stores an offset of the highest seen CDC mutation in a separate idx file 
> per commit log segment in cdc_raw. Clients tail this index file, delta their 
> local last parsed offset on change, and parse the corresponding commit log 
> segment using their last parsed offset as min
> * C* flags that index file with an offset and DONE when the file is flushed 
> so clients know when they can clean up
> h6. Details:
> * On creation of a CommitLogSegment, also hard-link the file in cdc_raw
> * On first write of a CDC-enabled mutation to a segment, we:
> ** Flag it as {{CDCState.CONTAINS}}
> ** Set a long tracking the {{CommitLogPosition}} of the 1st CDC-enabled 
> mutation in the log
> ** Set a long in the CommitLogSegment tracking the offset of the end of the 
> last written CDC mutation in the segment if higher than the previously known 
> highest CDC offset
> * On subsequent writes to the segment, we update the offset of the highest 
> known CDC data
> * On CommitLogSegment fsync, we write a file in cdc_raw as 
> _cdc.idx containing the min offset and end offset fsynced to 
> disk per file
> * On segment discard, if CDCState == {{CDCState.PERMITTED}}, delete both the 
> segment in commitlog and in cdc_raw
> * On segment discard, if CDCState == {{CDCState.CONTAINS}}, delete the 
> segment in commitlog and update the _cdc.idx file w/end offset 
> and a DONE marker
> * On segment replay, store the highest end offset of seen CDC-enabled 
> mutations from a segment and write that to _cdc.idx on 
> completion of segment replay. This should bridge the potential correctness 
> gap of a node writing to a segment and then dying before it can write the 
> _cdc.idx file.
> This should allow clients to skip the beginning of a file to the 1st CDC 
> mutation, track an offset of how far they've parsed, delta against the 
> _cdc.idx file end offset, and use that as a determinant on when to parse new 
> CDC data. Any existing clients written to the initial implementation of CDC 
> need only add the _cdc.idx logic and checking for DONE marker 
> to their code, so the burden on users to update to support this should be 
> quite small for the benefit of having data available as soon as it's fsynced 
> instead of at a non-deterministic time when potentially unrelated tables are 
> flushed.
> Finally, we should look into extending the interface on CommitLogReader to be 
> more friendly for realtime parsing, perhaps supporting taking a 
> CommitLogDescriptor and RandomAccessReader and resuming readSection calls, 
> assuming the reader is at the start of a SyncSegment. Would probably also 
> need to rewind to the start of the segment before returning so subsequent 
> calls would respect this contract. This would skip needing to deserialize the 
> descriptor and all completed SyncSegments to get to the root of the desired 
> segment for parsing.
> One alternative we discussed offline - instead of just storing the highest 
> seen CDC offset, we could instead store an offset per CDC mutation 
> (potentially delta encoded) in the idx file to allow clients to seek and only 
> parse the mutations with CDC enabled. My hunch is that the performance delta 
> from doing so wouldn't justify the complexity given the SyncSegment 
> deserialization and seeking restrictions in the compressed and encrypted 
> cases as mentioned above.
> The only complication I can think of with the above design is uncompressed 
> mmapped CommitLogSegments on Windows being undeletable, but it'd be pretty 
> simple to disallow configuration of CDC 

[jira] [Commented] (CASSANDRA-12148) Improve determinism of CDC data availability

[Joshua McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047120#comment-16047120
 ] 

Joshua McKenzie commented on CASSANDRA-12148:
-

It's on a back-burner but not forgotten. Still hoping to get it in by 4.0.

> Improve determinism of CDC data availability
> 
>
> Key: CASSANDRA-12148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12148
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Joshua McKenzie
>Assignee: Joshua McKenzie
> Fix For: 4.x
>
>
> The latency with which CDC data becomes available has a known limitation due 
> to our reliance on CommitLogSegments being discarded to have the data 
> available in cdc_raw: if a slowly written table co-habitates a 
> CommitLogSegment with CDC data, the CommitLogSegment won't be flushed until 
> we hit either memory pressure on memtables or CommitLog limit pressure. 
> Ultimately, this leaves a non-deterministic element to when data becomes 
> available for CDC consumption unless a consumer parses live CommitLogSegments.
> To work around this limitation and make semi-realtime CDC consumption more 
> friendly to end-users, I propose we extend CDC as follows:
> h6. High level:
> * Consumers parse hard links of active CommitLogSegments in cdc_raw instead 
> of waiting for flush/discard and file move
> * C* stores an offset of the highest seen CDC mutation in a separate idx file 
> per commit log segment in cdc_raw. Clients tail this index file, delta their 
> local last parsed offset on change, and parse the corresponding commit log 
> segment using their last parsed offset as min
> * C* flags that index file with an offset and DONE when the file is flushed 
> so clients know when they can clean up
> h6. Details:
> * On creation of a CommitLogSegment, also hard-link the file in cdc_raw
> * On first write of a CDC-enabled mutation to a segment, we:
> ** Flag it as {{CDCState.CONTAINS}}
> ** Set a long tracking the {{CommitLogPosition}} of the 1st CDC-enabled 
> mutation in the log
> ** Set a long in the CommitLogSegment tracking the offset of the end of the 
> last written CDC mutation in the segment if higher than the previously known 
> highest CDC offset
> * On subsequent writes to the segment, we update the offset of the highest 
> known CDC data
> * On CommitLogSegment fsync, we write a file in cdc_raw as 
> _cdc.idx containing the min offset and end offset fsynced to 
> disk per file
> * On segment discard, if CDCState == {{CDCState.PERMITTED}}, delete both the 
> segment in commitlog and in cdc_raw
> * On segment discard, if CDCState == {{CDCState.CONTAINS}}, delete the 
> segment in commitlog and update the _cdc.idx file w/end offset 
> and a DONE marker
> * On segment replay, store the highest end offset of seen CDC-enabled 
> mutations from a segment and write that to _cdc.idx on 
> completion of segment replay. This should bridge the potential correctness 
> gap of a node writing to a segment and then dying before it can write the 
> _cdc.idx file.
> This should allow clients to skip the beginning of a file to the 1st CDC 
> mutation, track an offset of how far they've parsed, delta against the 
> _cdc.idx file end offset, and use that as a determinant on when to parse new 
> CDC data. Any existing clients written to the initial implementation of CDC 
> need only add the _cdc.idx logic and checking for DONE marker 
> to their code, so the burden on users to update to support this should be 
> quite small for the benefit of having data available as soon as it's fsynced 
> instead of at a non-deterministic time when potentially unrelated tables are 
> flushed.
> Finally, we should look into extending the interface on CommitLogReader to be 
> more friendly for realtime parsing, perhaps supporting taking a 
> CommitLogDescriptor and RandomAccessReader and resuming readSection calls, 
> assuming the reader is at the start of a SyncSegment. Would probably also 
> need to rewind to the start of the segment before returning so subsequent 
> calls would respect this contract. This would skip needing to deserialize the 
> descriptor and all completed SyncSegments to get to the root of the desired 
> segment for parsing.
> One alternative we discussed offline - instead of just storing the highest 
> seen CDC offset, we could instead store an offset per CDC mutation 
> (potentially delta encoded) in the idx file to allow clients to seek and only 
> parse the mutations with CDC enabled. My hunch is that the performance delta 
> from doing so wouldn't justify the complexity given the SyncSegment 
> deserialization and seeking restrictions in the compressed and encrypted 
> cases as mentioned above.
> The only complication I can think of with the above design is uncompressed 
> mmapped CommitLogSegments 

[jira] [Comment Edited] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Jordan Beauchamp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047118#comment-16047118
 ] 

Jordan Beauchamp edited comment on CASSANDRA-13591 at 6/12/17 9:53 PM:
---

Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?

{quote} Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809
{quote}


was (Author: supremekai):
Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?

{quote} Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 T{quote}hread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may 

[jira] [Comment Edited] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Jordan Beauchamp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047118#comment-16047118
 ] 

Jordan Beauchamp edited comment on CASSANDRA-13591 at 6/12/17 9:52 PM:
---

Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?

{quote} Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 T{quote}hread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809


was (Author: supremekai):
Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?
{{
 Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809}}

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better 

[jira] [Comment Edited] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Jordan Beauchamp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047118#comment-16047118
 ] 

Jordan Beauchamp edited comment on CASSANDRA-13591 at 6/12/17 9:52 PM:
---

Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?
{{
 Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809}}


was (Author: supremekai):
Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?
```
 Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809
```

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better placed 

[jira] [Commented] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Jordan Beauchamp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047118#comment-16047118
 ] 

Jordan Beauchamp commented on CASSANDRA-13591:
--

Ran sstablescrub on a certain CF and received similar error:

[https://pastebin.com/1cumiMjB]

These errors seem to indicate a missing library?
```
 Internal exceptions (10 events):
 308 Event: 0.760 Thread 0x7f840800c000 Exception  (0xfe73b268) 
thrown at [/tmp/buildd/openjdk-8-8u121-b13/src/hots 
pot/src/share/vm/prims/methodHandles.cpp, line 1146]
 309 Event: 1.104 Thread 0x7f840800c000 Exception 
 (0x fc4b3300) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 li
 310 Event: 1.422 Thread 0x7f840800c000 Exception  
(0xfb368d80) thrown at [/tmp/buildd/openjdk-8-8u121-b 
13/src/hotspot/src/share/vm/prims/nativeLookup.cpp, line 378]
 311 Event: 1.422 Thread 0x7f840800c000 Exception 
 (0xf b370718) thrown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/prims/nativeLookup.cpp,
 line 378]
 312 Event: 1.967 Thread 0x7f840800c000 Exception  
(0xfb20fe68) thrown at [/tmp/buildd/openjdk-8-8u12 
1-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp, line 582]
 313 Event: 2.240 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8b89c75 to 0x7f83f8b8a019
 314 Event: 2.246 Thread 0x7f840800c000 Exception  
(0xfcba5788) th rown at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 582]
 315 Event: 2.258 Thread 0x7f840800c000 Exception  (0xfcc30368) thrown  at 
[/tmp/buildd/openjdk-8-8u121-b13/src/hotspot/src/share/vm/interpreter/linkResolver.cpp,
 line 641]
 316 Event: 7.999 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f95f201b to 0x7f83f95f24e5
 317 Event: 20.265 Thread 0x7f840800c000 Implicit null exception at 
0x7f83f8db56ea to 0x7f83f8db5809
```

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better placed at [https://github.com/lz4/lz4-java] but i'm 
> not sure how to get better details on the error.
> Potentially due to data corruption.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Jordan Beauchamp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047071#comment-16047071
 ] 

Jordan Beauchamp commented on CASSANDRA-13591:
--

I assume so, the node was running fine until this started happening.
I'm going to try manually sstablescrub every CF and will post results here

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better placed at [https://github.com/lz4/lz4-java] but i'm 
> not sure how to get better details on the error.
> Potentially due to data corruption.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13584) Inclusion of cassandra-dtest project from code donation

[Nate McCall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nate McCall resolved CASSANDRA-13584.
-
   Resolution: Fixed
Fix Version/s: 4.x

> Inclusion of cassandra-dtest project from code donation
> ---
>
> Key: CASSANDRA-13584
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13584
> Project: Cassandra
>  Issue Type: Task
>Reporter: Nate McCall
>Assignee: Nate McCall
> Fix For: 4.x
>
> Attachments: DataStax_software-grant.pdf
>
>
> The issue is for tracking legal and incubator acceptance of cassandra-dtest. 
> IP clearance template is located here:
> https://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/cassandra-dtest.xml
> The dtest project is located here:
> https://github.com/riptano/cassandra-dtest 
> For convienience, vote thread is here:
> https://lists.apache.org/thread.html/d9e694ba8eaac8e8c70cbfd3f6ee249d43f8c67279882ffc65e56cac@%3Cdev.cassandra.apache.org%3E
> Software grant from DataStax is attached. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046896#comment-16046896
 ] 

Dikang Gu commented on CASSANDRA-8735:
--

Hi, I'm not seeing this patch in trunk or latest 3.0.X, is it reverted or 
overwritten in some version? Thanks.

> Batch log replication is not randomized when there are only 2 racks
> ---
>
> Key: CASSANDRA-8735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8735
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Yuki Morishita
>Assignee: Mihai Suteu
>Priority: Minor
> Fix For: 2.1.9, 2.2.1, 3.0 alpha 1
>
> Attachments: 8735-v2.patch, CASSANDRA-8735.patch
>
>
> Batch log replication is not randomized and the same 2 nodes can be picked up 
> when there are only 2 racks in the cluster.
> https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046825#comment-16046825
 ] 

ASF GitHub Bot commented on CASSANDRA-13570:


Github user Jollyplum closed the pull request at:

https://github.com/apache/cassandra/pull/119


> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.0
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[Matt Byrd (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046826#comment-16046826
 ] 

Matt Byrd commented on CASSANDRA-13570:
---

Done, thanks

> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.0
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13593) Cassandra Service Crashes

[anuragh (JIRA)]

anuragh created CASSANDRA-13593:
---

 Summary: Cassandra Service Crashes
 Key: CASSANDRA-13593
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13593
 Project: Cassandra
  Issue Type: Bug
  Components: Configuration
 Environment: Cassandra Migration from one rack to another
Reporter: anuragh
Priority: Blocker
 Fix For: 2.1.18


I'm migrating from one rack to another.
Source Nodes 3,each node contains 175GB of data
Target Nodes 3
Add one of the ipaddress to the seed node .
existing 3 nodes from the source + newly added node in one ring.
The moment i added to the ring,cassandra service crashes




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[Joel Knighton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-11381:
--
Status: Ready to Commit  (was: Patch Available)

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13591) LZ4 Decompression during startup crashes

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046609#comment-16046609
 ] 

Stefan Podkowinski commented on CASSANDRA-13591:


Does this only happen during compactions? Can you identify any corrupt sstable 
files that would cause this?

> LZ4 Decompression during startup crashes
> 
>
> Key: CASSANDRA-13591
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13591
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: I've received this bug using 3.05 and 3.10.
> Running C* in docker container using cassandra:3.5 and cassandra:3.10 issues 
> on Ubuntu 16.04
> {code:bash}
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 
> 1.8.0_121-8u121-b13-1~bpo8+1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 
> compressed oops)
> {code}
>Reporter: Jordan Beauchamp
>Priority: Critical
> Fix For: 3.0.x
>
> Attachments: hs_err_pid1.log
>
>
> C* docker container is in crash loop.
> Stack trace seems to implicate lz4 fast decompress during compaction task.
> This issue may be better placed at [https://github.com/lz4/lz4-java] but i'm 
> not sure how to get better details on the error.
> Potentially due to data corruption.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046602#comment-16046602
 ] 

Joel Knighton commented on CASSANDRA-11381:
---

Great! I also ran dtests on some other machines this weekend and they looked 
good on all branches (relative to the current state of the branches).

+1 - one minor nit that can be fixed on commit: new CHANGES.txt entries should 
go at the top of the list.

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13592) Null Pointer exception at SELECT JSON statement

[Wyss Philipp (JIRA)]

Wyss Philipp created CASSANDRA-13592:


 Summary: Null Pointer exception at SELECT JSON statement
 Key: CASSANDRA-13592
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13592
 Project: Cassandra
  Issue Type: Bug
  Components: CQL
 Environment: Debian Linux
Reporter: Wyss Philipp
 Attachments: system.log

A Nulll pointer exception appears when the command
SELECT JSON * FROM examples.basic;

---MORE---


Examples.basic has the following description (DESC examples.basic;):
CREATE TABLE examples.basic (
key frozen> PRIMARY KEY,
wert text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

The error appears after the ---MORE--- line.

The field "wert" has a JSON formatted string.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046491#comment-16046491
 ] 

Marcus Eriksson commented on CASSANDRA-13570:
-

crap, forgot the "This closes #119" - I think you have to close the PR

> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.0
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13570:

   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

+1

Committed

> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.0
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Allow sub-range repairs (specifying -et -st) for a preview of repaired data

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/trunk ff29d6099 -> 3e8f3c5ed


Allow sub-range repairs (specifying -et -st) for a preview of repaired data

Patch by Matt Byrd; reviewed by marcuse for CASSANDRA-13570


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3e8f3c5e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3e8f3c5e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3e8f3c5e

Branch: refs/heads/trunk
Commit: 3e8f3c5ed06242a6aedf76c216b1c48c105f84e6
Parents: ff29d60
Author: Matt Byrd 
Authored: Thu Jun 8 16:21:19 2017 -0700
Committer: Marcus Eriksson 
Committed: Mon Jun 12 14:15:23 2017 +0200

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/repair/messages/RepairOption.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3e8f3c5e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4425aad..291d93a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Allow sub-range repairs for a preview of repaired data (CASSANDRA-13570)
  * NPE in IR cleanup when columnfamily has no sstables (CASSANDRA-13585)
  * Fix Randomness of stress values (CASSANDRA-12744)
  * Allow selecting Map values and Set elements (CASSANDRA-7396)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3e8f3c5e/src/java/org/apache/cassandra/repair/messages/RepairOption.java
--
diff --git a/src/java/org/apache/cassandra/repair/messages/RepairOption.java 
b/src/java/org/apache/cassandra/repair/messages/RepairOption.java
index b6f125a..6d69cf0 100644
--- a/src/java/org/apache/cassandra/repair/messages/RepairOption.java
+++ b/src/java/org/apache/cassandra/repair/messages/RepairOption.java
@@ -244,7 +244,7 @@ public class RepairOption
 }
 }
 
-if (option.isIncremental() && !option.isGlobal())
+if (option.isIncremental() && !option.isPreview() && 
!option.isGlobal())
 {
 throw new IllegalArgumentException("Incremental repairs cannot be 
run against a subset of tokens or ranges");
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13570:

Reviewer: Marcus Eriksson

> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.x
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13570) allow sub-range repairs (specifying -et -st) for a preview of repaired data

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13570:

Status: Patch Available  (was: Open)

> allow sub-range repairs (specifying -et -st) for a preview of repaired data
> ---
>
> Key: CASSANDRA-13570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13570
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Byrd
>Assignee: Matt Byrd
>Priority: Minor
> Fix For: 4.x
>
>
> I don't see any inherent reason for restricting preview repairs of repaired 
> data to not allow specifying start and end tokens. 
> The restriction seems to be coming from the fact that incremental=true in 
> RepairOption, which is the case but it's not truly an incremental repair 
> since we're only previewing.
> {code:java}
> if (option.isIncremental() && !option.isGlobal())
> {
> throw new IllegalArgumentException("Incremental repairs cannot be 
> run against a subset of tokens or ranges");
> }
> {code}
> It would be helpful to allow this, so that operators could sequence a sweep 
> over the entirety of the token-space in a more gradual fashion.
> Also it might help in examining which portions of the token-space differ.
> Can anyone see any reasons for not allowing this?
> I.e just changing the above to something like:
> {code:java}
>  if (option.isIncremental() && !option.getPreviewKind().isPreview() && 
> !option.isGlobal())
>   {
>   throw new IllegalArgumentException("Incremental repairs cannot 
> be run against a subset of tokens or ranges");
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13575) Snapshot fails on IndexInfo

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046281#comment-16046281
 ] 

Hannu Kröger commented on CASSANDRA-13575:
--

Restarting seems to fix the issue. At least temporarily.

> Snapshot fails on IndexInfo
> ---
>
> Key: CASSANDRA-13575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13575
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>
> Snapshot creation fails on IndexInfo table. This has happened in several 
> Cassandra environments.
> There is also Stratio lucene index installed 2.2.3.1. I don't know if that 
> matters.
> {code}
> [root@host1 IndexInfo-9f5c6374d48532299a0a5094af9ad1e3]# nodetool snapshot -t 
> testsnapshot
> Requested creating snapshot(s) for [all keyspaces] with snapshot name 
> [testsnapshot]
> error: Tried to hard link to file that does not exist 
> /cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/la-264-big-Filter.db
> -- StackTrace --
> java.lang.RuntimeException: Tried to hard link to file that does not exist 
> /cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/la-264-big-Filter.db
> at 
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:85)
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1763)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2328)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2453)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2443)
> at org.apache.cassandra.db.Keyspace.snapshot(Keyspace.java:198)
> at 
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2604)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at 
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at 
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
>