[jira] [Commented] (CASSANDRA-13464) Failed to create Materialized view with a specific token range

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136337#comment-16136337
 ] 

ZhaoYang commented on CASSANDRA-13464:
--

Thanks for the patch. The fix looks good.

A few nits:

1. the assertion in View.java maybe not necessary.
2. In test, there is existing {{assertInvalidThrowMessage}} function. And good 
to put those tests cases into ViewSchemaTest.

> Failed to create Materialized view with a specific token range
> --
>
> Key: CASSANDRA-13464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13464
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Natsumi Kojima
>Assignee: Krishna Dattu Koneru
>Priority: Minor
>  Labels: materializedviews
>
> Failed to create Materialized view with a specific token range.
> Example :
> {code:java}
> $ ccm create "MaterializedView" -v 3.0.13
> $ ccm populate  -n 3
> $ ccm start
> $ ccm status
> Cluster: 'MaterializedView'
> ---
> node1: UP
> node3: UP
> node2: UP
> $ccm node1 cqlsh
> Connected to MaterializedView at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.0.13 | CQL spec 3.4.0 | Native protocol v4]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE test WITH replication = {'class':'SimpleStrategy', 
> 'replication_factor':3};
> cqlsh> CREATE TABLE test.test ( id text PRIMARY KEY , value1 text , value2 
> text, value3 text);
> $ccm node1 ring test 
> Datacenter: datacenter1
> ==
> AddressRackStatus State   LoadOwns
> Token
>   
> 3074457345618258602
> 127.0.0.1  rack1   Up Normal  64.86 KB100.00% 
> -9223372036854775808
> 127.0.0.2  rack1   Up Normal  86.49 KB100.00% 
> -3074457345618258603
> 127.0.0.3  rack1   Up Normal  89.04 KB100.00% 
> 3074457345618258602
> $ ccm node1 cqlsh
> cqlsh> INSERT INTO test.test (id, value1 , value2, value3 ) VALUES ('aaa', 
> 'aaa', 'aaa' ,'aaa');
> cqlsh> INSERT INTO test.test (id, value1 , value2, value3 ) VALUES ('bbb', 
> 'bbb', 'bbb' ,'bbb');
> cqlsh> SELECT token(id),id,value1 FROM test.test;
>  system.token(id) | id  | value1
> --+-+
>  -4737872923231490581 | aaa |aaa
>  -3071845237020185195 | bbb |bbb
> (2 rows)
> cqlsh> CREATE MATERIALIZED VIEW test.test_view AS SELECT value1, id FROM 
> test.test WHERE id IS NOT NULL AND value1 IS NOT NULL AND TOKEN(id) > 
> -9223372036854775808 AND TOKEN(id) < -3074457345618258603 PRIMARY KEY(value1, 
> id) WITH CLUSTERING ORDER BY (id ASC);
> ServerError: java.lang.ClassCastException: 
> org.apache.cassandra.cql3.TokenRelation cannot be cast to 
> org.apache.cassandra.cql3.SingleColumnRelation
> {code}
> Stacktrace :
> {code:java}
> INFO  [MigrationStage:1] 2017-04-19 18:32:48,131 ColumnFamilyStore.java:389 - 
> Initializing test.test
> WARN  [SharedPool-Worker-1] 2017-04-19 18:44:07,263 FBUtilities.java:337 - 
> Trigger directory doesn't exist, please create it and try again.
> ERROR [SharedPool-Worker-1] 2017-04-19 18:46:10,072 QueryMessage.java:128 - 
> Unexpected error during query
> java.lang.ClassCastException: org.apache.cassandra.cql3.TokenRelation cannot 
> be cast to org.apache.cassandra.cql3.SingleColumnRelation
>   at 
> org.apache.cassandra.db.view.View.relationsToWhereClause(View.java:275) 
> ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.cql3.statements.CreateViewStatement.announceMigration(CreateViewStatement.java:219)
>  ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93)
>  ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) 
> ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) 
> ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  ~[apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  [apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  [apache-cassandra-3.0.13.jar:3.0.13]
>   at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.44.Final.jar:4.0.44.Final]
>   at 
> 

[jira] [Created] (CASSANDRA-13783) Updating the plugin url link

[Amitkumar Ghatwal (JIRA)]

Amitkumar Ghatwal created CASSANDRA-13783:
-

 Summary: Updating the plugin url link
 Key: CASSANDRA-13783
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13783
 Project: Cassandra
  Issue Type: Task
  Components: Documentation and Website
Reporter: Amitkumar Ghatwal
Priority: Critical
 Fix For: 4.x


Hi All,
 
Updating index.rst with url of plugin

Moved the capi-row cache plugin github repository from an individual github 
account to a more generic one.
Older plugin repo location : https://github.com/hhorii/capi-rowcache
New plugin repo location : https://github.com/ppc64le/capi-rowcache

PR link : 
https://github.com/apache/cassandra/pull/139/commits/05851cdabc953ec17480b0bd325b1b5adb7b53bb

Reference : https://issues.apache.org/jira/browse/CASSANDRA-13581




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13773) cassandra-stress writes even data when n=0

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13773:
-
   Resolution: Fixed
Fix Version/s: 4.0
   3.11.1
   Status: Resolved  (was: Patch Available)

dtests looked good as well.

I don't think anyone would expect cassandra-stress to write data if n=0, so 
I've committed it to 3.0 as {{6a1b1f26b7174e8c9bf86a96514ab626ce2a4117}} and 
merged into 3.11 and trunk.

> cassandra-stress writes even data when n=0
> --
>
> Key: CASSANDRA-13773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13773
> Project: Cassandra
>  Issue Type: Bug
>  Components: Stress
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
>Priority: Minor
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> This is very unintuitive as
> {code}
> cassandra-stress write n=0 -rate threads=1
> {code}
> will do inserts even with *n=0*. I guess most people won't ever run with 
> *n=0* but this is a nice shortcut for creating some schema without using 
> *cqlsh*
> This is happening because we're writing *50k* rows of warmup data as can be 
> seen below:
> {code}
> cqlsh> select count(*) from keyspace1.standard1 ;
>  count
> ---
>  5
> (1 rows)
> {code}
> We can avoid writing warmup data using 
> {code}
> cassandra-stress write n=0 no-warmup -rate threads=1
> {code}
> but I would still expect to have *0* rows written when specifying *n=0*.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[stefania]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d4a7e7b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d4a7e7b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d4a7e7b

Branch: refs/heads/trunk
Commit: 3d4a7e7b61c7eb2c8736fce2bde47c14564d5982
Parents: 97b1c3c 59d4c27
Author: Stefania Alborghetti 
Authored: Tue Aug 22 09:36:23 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:36:23 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d4a7e7b/CHANGES.txt
--
diff --cc CHANGES.txt
index 59b401c,9e42ffb..75a4be9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -126,6 -6,8 +126,7 @@@
   * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
   * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
  Merged from 3.0:
+  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 - * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)
   * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d4a7e7b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Don't let stress write warmup data if n=0

[stefania]

Don't let stress write warmup data if n=0

patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for 
CASSANDRA-13773


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6a1b1f26
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6a1b1f26
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6a1b1f26

Branch: refs/heads/trunk
Commit: 6a1b1f26b7174e8c9bf86a96514ab626ce2a4117
Parents: ec85b4a
Author: Eduard Tudenhoefner 
Authored: Mon Aug 21 11:11:00 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:28:00 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8b22f0..97dda05 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java 
b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index cda54a0..8b15e92 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -54,6 +54,13 @@ public class StressAction implements Runnable
 // creating keyspace and column families
 settings.maybeCreateKeyspaces();
 
+if (settings.command.count == 0)
+{
+output.println("N=0: SCHEMA CREATED, NOTHING ELSE DONE.");
+settings.disconnect();
+return;
+}
+
 output.println("Sleeping 2s...");
 Uninterruptibles.sleepUninterruptibly(2, TimeUnit.SECONDS);
 
@@ -87,9 +94,7 @@ public class StressAction implements Runnable
 {
 PrintStream warmupOutput = new PrintStream(new OutputStream() { 
@Override public void write(int b) throws IOException { } } );
 // do 25% of iterations as warmup but no more than 50k (by default 
hotspot compiles methods after 10k invocations)
-int iterations = (settings.command.count > 0
- ? Math.min(5, (int)(settings.command.count * 
0.25))
- : 5) * settings.node.nodes.size();
+int iterations = Math.min(5, (int) (settings.command.count * 
0.25)) * settings.node.nodes.size();
 int threads = 100;
 
 if (settings.rate.maxThreads > 0)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Don't let stress write warmup data if n=0

[stefania]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 ec85b4a96 -> 6a1b1f26b
  refs/heads/cassandra-3.11 7f8417c40 -> 59d4c2719
  refs/heads/trunk 97b1c3ce8 -> 3d4a7e7b6


Don't let stress write warmup data if n=0

patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for 
CASSANDRA-13773


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6a1b1f26
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6a1b1f26
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6a1b1f26

Branch: refs/heads/cassandra-3.0
Commit: 6a1b1f26b7174e8c9bf86a96514ab626ce2a4117
Parents: ec85b4a
Author: Eduard Tudenhoefner 
Authored: Mon Aug 21 11:11:00 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:28:00 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8b22f0..97dda05 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java 
b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index cda54a0..8b15e92 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -54,6 +54,13 @@ public class StressAction implements Runnable
 // creating keyspace and column families
 settings.maybeCreateKeyspaces();
 
+if (settings.command.count == 0)
+{
+output.println("N=0: SCHEMA CREATED, NOTHING ELSE DONE.");
+settings.disconnect();
+return;
+}
+
 output.println("Sleeping 2s...");
 Uninterruptibles.sleepUninterruptibly(2, TimeUnit.SECONDS);
 
@@ -87,9 +94,7 @@ public class StressAction implements Runnable
 {
 PrintStream warmupOutput = new PrintStream(new OutputStream() { 
@Override public void write(int b) throws IOException { } } );
 // do 25% of iterations as warmup but no more than 50k (by default 
hotspot compiles methods after 10k invocations)
-int iterations = (settings.command.count > 0
- ? Math.min(5, (int)(settings.command.count * 
0.25))
- : 5) * settings.node.nodes.size();
+int iterations = Math.min(5, (int) (settings.command.count * 
0.25)) * settings.node.nodes.size();
 int threads = 100;
 
 if (settings.rate.maxThreads > 0)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[stefania]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/59d4c271
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/59d4c271
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/59d4c271

Branch: refs/heads/trunk
Commit: 59d4c27194c06d593523f40fa2fca819ef5046a4
Parents: 7f8417c 6a1b1f2
Author: Stefania Alborghetti 
Authored: Tue Aug 22 09:33:07 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:33:07 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/59d4c271/CHANGES.txt
--
diff --cc CHANGES.txt
index 0520477,97dda05..9e42ffb
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
   * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/59d4c271/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --cc tools/stress/src/org/apache/cassandra/stress/StressAction.java
index 5a340e8,8b15e92..670c187
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@@ -94,13 -90,11 +101,11 @@@ public class StressAction implements Ru
  }
  
  // type provided separately to support recursive call for mixed command 
with each command type it is performing
 +@SuppressWarnings("resource") // warmupOutput doesn't need closing
  private void warmup(OpDistributionFactory operations)
  {
 -PrintStream warmupOutput = new PrintStream(new OutputStream() { 
@Override public void write(int b) throws IOException { } } );
  // do 25% of iterations as warmup but no more than 50k (by default 
hotspot compiles methods after 10k invocations)
- int iterations = (settings.command.count > 0
-  ? Math.min(5, (int)(settings.command.count * 
0.25))
-  : 5) * settings.node.nodes.size();
+ int iterations = Math.min(5, (int) (settings.command.count * 
0.25)) * settings.node.nodes.size();
  int threads = 100;
  
  if (settings.rate.maxThreads > 0)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Don't let stress write warmup data if n=0

[stefania]

Don't let stress write warmup data if n=0

patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for 
CASSANDRA-13773


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6a1b1f26
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6a1b1f26
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6a1b1f26

Branch: refs/heads/cassandra-3.11
Commit: 6a1b1f26b7174e8c9bf86a96514ab626ce2a4117
Parents: ec85b4a
Author: Eduard Tudenhoefner 
Authored: Mon Aug 21 11:11:00 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:28:00 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8b22f0..97dda05 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a1b1f26/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java 
b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index cda54a0..8b15e92 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -54,6 +54,13 @@ public class StressAction implements Runnable
 // creating keyspace and column families
 settings.maybeCreateKeyspaces();
 
+if (settings.command.count == 0)
+{
+output.println("N=0: SCHEMA CREATED, NOTHING ELSE DONE.");
+settings.disconnect();
+return;
+}
+
 output.println("Sleeping 2s...");
 Uninterruptibles.sleepUninterruptibly(2, TimeUnit.SECONDS);
 
@@ -87,9 +94,7 @@ public class StressAction implements Runnable
 {
 PrintStream warmupOutput = new PrintStream(new OutputStream() { 
@Override public void write(int b) throws IOException { } } );
 // do 25% of iterations as warmup but no more than 50k (by default 
hotspot compiles methods after 10k invocations)
-int iterations = (settings.command.count > 0
- ? Math.min(5, (int)(settings.command.count * 
0.25))
- : 5) * settings.node.nodes.size();
+int iterations = Math.min(5, (int) (settings.command.count * 
0.25)) * settings.node.nodes.size();
 int threads = 100;
 
 if (settings.rate.maxThreads > 0)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[stefania]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/59d4c271
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/59d4c271
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/59d4c271

Branch: refs/heads/cassandra-3.11
Commit: 59d4c27194c06d593523f40fa2fca819ef5046a4
Parents: 7f8417c 6a1b1f2
Author: Stefania Alborghetti 
Authored: Tue Aug 22 09:33:07 2017 +0800
Committer: Stefania Alborghetti 
Committed: Tue Aug 22 09:33:07 2017 +0800

--
 CHANGES.txt  |  1 +
 .../src/org/apache/cassandra/stress/StressAction.java| 11 ---
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/59d4c271/CHANGES.txt
--
diff --cc CHANGES.txt
index 0520477,97dda05..9e42ffb
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
   * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/59d4c271/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --cc tools/stress/src/org/apache/cassandra/stress/StressAction.java
index 5a340e8,8b15e92..670c187
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@@ -94,13 -90,11 +101,11 @@@ public class StressAction implements Ru
  }
  
  // type provided separately to support recursive call for mixed command 
with each command type it is performing
 +@SuppressWarnings("resource") // warmupOutput doesn't need closing
  private void warmup(OpDistributionFactory operations)
  {
 -PrintStream warmupOutput = new PrintStream(new OutputStream() { 
@Override public void write(int b) throws IOException { } } );
  // do 25% of iterations as warmup but no more than 50k (by default 
hotspot compiles methods after 10k invocations)
- int iterations = (settings.command.count > 0
-  ? Math.min(5, (int)(settings.command.count * 
0.25))
-  : 5) * settings.node.nodes.size();
+ int iterations = Math.min(5, (int) (settings.command.count * 
0.25)) * settings.node.nodes.size();
  int threads = 100;
  
  if (settings.rate.maxThreads > 0)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13578) mx4j configuration minor improvement

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136103#comment-16136103
 ] 

Jeremiah Jordan edited comment on CASSANDRA-13578 at 8/22/17 12:42 AM:
---

For anyone that knew how this was supposed to be set this is changing the 
interface of using MX4J_PORT and MX4J_ADDRESS.  If you want to introduce new 
options that just take a port and address then I think it would be more upgrade 
friendly to give them new names, and then accept the old way if set, or if not 
try the new way.


was (Author: jjordan):
For anyone that knew how is was supposed to be set this is changing the 
interface of using MX4J_PORT and MX4J_ADDRESS.  If you want to introduce new 
options that just take a port and address then I think it would be more upgrade 
friendly to give them new names, and then accept the old way if set, or if not 
try the new way.

> mx4j configuration minor improvement
> 
>
> Key: CASSANDRA-13578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13578
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 4.x
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13578) mx4j configuration minor improvement

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136103#comment-16136103
 ] 

Jeremiah Jordan commented on CASSANDRA-13578:
-

For anyone that knew how is was supposed to be set this is changing the 
interface of using MX4J_PORT and MX4J_ADDRESS.  If you want to introduce new 
options that just take a port and address then I think it would be more upgrade 
friendly to give them new names, and then accept the old way if set, or if not 
try the new way.

> mx4j configuration minor improvement
> 
>
> Key: CASSANDRA-13578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13578
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 4.x
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-13662:

Reviewer: Jeremiah Jordan

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136084#comment-16136084
 ] 

Jeremiah Jordan commented on CASSANDRA-13662:
-

+1 Code LGTM.

I was looking at this the other day and wondering why it was still there.

Should we have a test showing that sending this returns the error from 
UnsupportedMessageCodec?

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13642) Expose recent histograms in JmxHistograms

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-13642:

Status: Ready to Commit  (was: Patch Available)

> Expose recent histograms in JmxHistograms
> -
>
> Key: CASSANDRA-13642
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13642
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>
> For monitoring tools that were written for the recent*Histograms the current 
> decaying and all time values exposed by the JmxHistograms is not consumable. 
> We can add a new attribute (easier with some tooling than operations to read) 
> to expose it just like in storage service previously.
> Should additionally make it so this attribute only stores previous values if 
> read, so that it does not add any additional memory cost to C* unless used 
> since we already storing 2 versions of histogram for the decaying/alltime 
> views.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13642) Expose recent histograms in JmxHistograms

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136069#comment-16136069
 ] 

Jeremiah Jordan commented on CASSANDRA-13642:
-

Looks good to me.  Definitely have written tools based on this behavior in the 
past.

> Expose recent histograms in JmxHistograms
> -
>
> Key: CASSANDRA-13642
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13642
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>
> For monitoring tools that were written for the recent*Histograms the current 
> decaying and all time values exposed by the JmxHistograms is not consumable. 
> We can add a new attribute (easier with some tooling than operations to read) 
> to expose it just like in storage service previously.
> Should additionally make it so this attribute only stores previous values if 
> read, so that it does not add any additional memory cost to C* unless used 
> since we already storing 2 versions of histogram for the decaying/alltime 
> views.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13642) Expose recent histograms in JmxHistograms

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-13642:

Reviewer: Jeremiah Jordan

> Expose recent histograms in JmxHistograms
> -
>
> Key: CASSANDRA-13642
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13642
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>
> For monitoring tools that were written for the recent*Histograms the current 
> decaying and all time values exposed by the JmxHistograms is not consumable. 
> We can add a new attribute (easier with some tooling than operations to read) 
> to expose it just like in storage service previously.
> Should additionally make it so this attribute only stores previous values if 
> read, so that it does not add any additional memory cost to C* unless used 
> since we already storing 2 versions of histogram for the decaying/alltime 
> views.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-12627) Provide new seed providers

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-12627.
-
Resolution: Won't Fix

> Provide new seed providers
> --
>
> Key: CASSANDRA-12627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12627
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Edward Capriolo
>
> SeedProvider is plugable, however only one implementation exists.
> Changes:
> * Create a SeedProvider that reads properties from System properties or env
> * Provide a SeedProvider that scans ranges of IP addresses to find peers.
> * Refactor interface to abstract class because all seed providers must 
> provide a constructor that accepts Map 
> * correct error messages
> * Do not catch Exception use MultiCatch and catch typed exceptions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-12966:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Gossip thread slows down when using batch commit log
> 
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> When using batch commit log mode, the Gossip thread slows down when peers 
> after a node bounces. This is because we perform a bunch of updates to the 
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized 
> method. How quickly each one of those individual updates takes depends on how 
> busy the system is at the time wrt write traffic. If the system is largely 
> quiescent, each update will be relatively quick (just waiting for the fsync). 
> If the system is getting a lot of writes, and depending on the 
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can 
> get stuck in the backlog, which causes the Gossip thread to stop processing. 
> We have observed in large clusters that a rolling restart causes triggers and 
> exacerbates this behavior. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12966) Gossip thread slows down when using batch commit log

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135984#comment-16135984
 ] 

Jason Brown commented on CASSANDRA-12966:
-

After many months, I finally rebased and committed (tests still passed). commit 
sha is {{ec85b4a96390ef5bf85acac42f9c93a68620b668}}

> Gossip thread slows down when using batch commit log
> 
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> When using batch commit log mode, the Gossip thread slows down when peers 
> after a node bounces. This is because we perform a bunch of updates to the 
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized 
> method. How quickly each one of those individual updates takes depends on how 
> busy the system is at the time wrt write traffic. If the system is largely 
> quiescent, each update will be relatively quick (just waiting for the fsync). 
> If the system is getting a lot of writes, and depending on the 
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can 
> get stuck in the backlog, which causes the Gossip thread to stop processing. 
> We have observed in large clusters that a rolling restart causes triggers and 
> exacerbates this behavior. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f8417c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f8417c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f8417c4

Branch: refs/heads/cassandra-3.11
Commit: 7f8417c400ff36be15d7b4d2a5a64706f090e288
Parents: b726f26 ec85b4a
Author: Jason Brown 
Authored: Mon Aug 21 15:49:38 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:53:31 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java | 22 +--
 .../cassandra/service/StorageService.java   | 28 +++-
 .../apache/cassandra/db/SystemKeyspaceTest.java |  6 -
 .../cassandra/gms/FailureDetectorTest.java  |  2 +-
 .../service/LeaveAndBootstrapTest.java  |  9 +--
 6 files changed, 43 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/CHANGES.txt
--
diff --cc CHANGES.txt
index 7d20ab3,d8b22f0..0520477
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)
   * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/src/java/org/apache/cassandra/db/SystemKeyspace.java
--
diff --cc src/java/org/apache/cassandra/db/SystemKeyspace.java
index 6c45329,7ce74a1..ac51662
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@@ -28,19 -29,21 +29,24 @@@ import java.util.stream.Collectors
  import java.util.stream.StreamSupport;
  import javax.management.openmbean.OpenDataException;
  import javax.management.openmbean.TabularData;
+ import java.util.concurrent.Future;
  
  import com.google.common.collect.HashMultimap;
 +import com.google.common.collect.ImmutableMap;
  import com.google.common.collect.ImmutableSet;
  import com.google.common.collect.SetMultimap;
 +import com.google.common.collect.Sets;
  import com.google.common.io.ByteStreams;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
+ import com.google.common.util.concurrent.Futures;
+ 
+ import org.apache.cassandra.concurrent.Stage;
+ import org.apache.cassandra.concurrent.StageManager;
  import org.apache.cassandra.config.CFMetaData;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.config.SchemaConstants;
  import org.apache.cassandra.cql3.QueryProcessor;
  import org.apache.cassandra.cql3.UntypedResultSet;
  import org.apache.cassandra.cql3.functions.*;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/src/java/org/apache/cassandra/service/StorageService.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
--
diff --cc test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
index 256569f,d151f59..92f2a56
--- a/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
+++ b/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
@@@ -29,8 -30,9 +30,10 @@@ import org.apache.commons.io.FileUtils
  import org.junit.BeforeClass;
  import org.junit.Test;
  
+ import org.apache.cassandra.concurrent.Stage;
+ import org.apache.cassandra.concurrent.StageManager;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.config.SchemaConstants;
  import org.apache.cassandra.cql3.QueryProcessor;
  import org.apache.cassandra.cql3.UntypedResultSet;
  import org.apache.cassandra.dht.ByteOrderedPartitioner.BytesToken;

[3/6] cassandra git commit: Gossip thread slows down when using batch commit log

[jasobrown]

Gossip thread slows down when using batch commit log

patch by jasobrown; reviwed by spodkowinski fot CASSANDRA-12966


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ec85b4a9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ec85b4a9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ec85b4a9

Branch: refs/heads/trunk
Commit: ec85b4a96390ef5bf85acac42f9c93a68620b668
Parents: dc32ed8
Author: Jason Brown 
Authored: Mon Nov 28 15:22:14 2016 -0800
Committer: Jason Brown 
Committed: Mon Aug 21 15:48:09 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java | 22 +--
 .../cassandra/service/StorageService.java   | 28 +++-
 .../apache/cassandra/db/SystemKeyspaceTest.java |  6 -
 .../cassandra/gms/FailureDetectorTest.java  |  2 +-
 .../service/LeaveAndBootstrapTest.java  | 10 +--
 6 files changed, 44 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a0a61ac..d8b22f0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)
  * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/src/java/org/apache/cassandra/db/SystemKeyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/SystemKeyspace.java 
b/src/java/org/apache/cassandra/db/SystemKeyspace.java
index cc21435..7ce74a1 100644
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@ -23,11 +23,13 @@ import java.io.IOException;
 import java.net.InetAddress;
 import java.nio.ByteBuffer;
 import java.util.*;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import java.util.stream.StreamSupport;
 import javax.management.openmbean.OpenDataException;
 import javax.management.openmbean.TabularData;
+import java.util.concurrent.Future;
 
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableSet;
@@ -36,6 +38,10 @@ import com.google.common.io.ByteStreams;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.util.concurrent.Futures;
+
+import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.QueryProcessor;
@@ -687,29 +693,29 @@ public final class SystemKeyspace
 /**
  * Record tokens being used by another node
  */
-public static synchronized void updateTokens(InetAddress ep, 
Collection tokens)
+public static Future updateTokens(final InetAddress ep, final 
Collection tokens, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return Futures.immediateFuture(null);
 
 String req = "INSERT INTO system.%s (peer, tokens) VALUES (?, ?)";
-executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens));
+return executorService.submit((Runnable) () -> 
executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens)));
 }
 
-public static synchronized void updatePreferredIP(InetAddress ep, 
InetAddress preferred_ip)
+public static void updatePreferredIP(InetAddress ep, InetAddress 
preferred_ip)
 {
 String req = "INSERT INTO system.%s (peer, preferred_ip) VALUES (?, 
?)";
 executeInternal(String.format(req, PEERS), ep, preferred_ip);
 forceBlockingFlush(PEERS);
 }
 
-public static synchronized void updatePeerInfo(InetAddress ep, String 
columnName, Object value)
+public static Future updatePeerInfo(final InetAddress ep, final String 
columnName, final Object value, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return Futures.immediateFuture(null);
 
 String req = "INSERT INTO system.%s (peer, %s) VALUES (?, ?)";
-executeInternal(String.format(req, PEERS, columnName), ep, value);
+return 

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f8417c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f8417c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f8417c4

Branch: refs/heads/trunk
Commit: 7f8417c400ff36be15d7b4d2a5a64706f090e288
Parents: b726f26 ec85b4a
Author: Jason Brown 
Authored: Mon Aug 21 15:49:38 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:53:31 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java | 22 +--
 .../cassandra/service/StorageService.java   | 28 +++-
 .../apache/cassandra/db/SystemKeyspaceTest.java |  6 -
 .../cassandra/gms/FailureDetectorTest.java  |  2 +-
 .../service/LeaveAndBootstrapTest.java  |  9 +--
 6 files changed, 43 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/CHANGES.txt
--
diff --cc CHANGES.txt
index 7d20ab3,d8b22f0..0520477
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)
   * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/src/java/org/apache/cassandra/db/SystemKeyspace.java
--
diff --cc src/java/org/apache/cassandra/db/SystemKeyspace.java
index 6c45329,7ce74a1..ac51662
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@@ -28,19 -29,21 +29,24 @@@ import java.util.stream.Collectors
  import java.util.stream.StreamSupport;
  import javax.management.openmbean.OpenDataException;
  import javax.management.openmbean.TabularData;
+ import java.util.concurrent.Future;
  
  import com.google.common.collect.HashMultimap;
 +import com.google.common.collect.ImmutableMap;
  import com.google.common.collect.ImmutableSet;
  import com.google.common.collect.SetMultimap;
 +import com.google.common.collect.Sets;
  import com.google.common.io.ByteStreams;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
+ import com.google.common.util.concurrent.Futures;
+ 
+ import org.apache.cassandra.concurrent.Stage;
+ import org.apache.cassandra.concurrent.StageManager;
  import org.apache.cassandra.config.CFMetaData;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.config.SchemaConstants;
  import org.apache.cassandra.cql3.QueryProcessor;
  import org.apache.cassandra.cql3.UntypedResultSet;
  import org.apache.cassandra.cql3.functions.*;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/src/java/org/apache/cassandra/service/StorageService.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f8417c4/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
--
diff --cc test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
index 256569f,d151f59..92f2a56
--- a/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
+++ b/test/unit/org/apache/cassandra/db/SystemKeyspaceTest.java
@@@ -29,8 -30,9 +30,10 @@@ import org.apache.commons.io.FileUtils
  import org.junit.BeforeClass;
  import org.junit.Test;
  
+ import org.apache.cassandra.concurrent.Stage;
+ import org.apache.cassandra.concurrent.StageManager;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.config.SchemaConstants;
  import org.apache.cassandra.cql3.QueryProcessor;
  import org.apache.cassandra.cql3.UntypedResultSet;
  import org.apache.cassandra.dht.ByteOrderedPartitioner.BytesToken;

[2/6] cassandra git commit: Gossip thread slows down when using batch commit log

[jasobrown]

Gossip thread slows down when using batch commit log

patch by jasobrown; reviwed by spodkowinski fot CASSANDRA-12966


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ec85b4a9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ec85b4a9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ec85b4a9

Branch: refs/heads/cassandra-3.11
Commit: ec85b4a96390ef5bf85acac42f9c93a68620b668
Parents: dc32ed8
Author: Jason Brown 
Authored: Mon Nov 28 15:22:14 2016 -0800
Committer: Jason Brown 
Committed: Mon Aug 21 15:48:09 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java | 22 +--
 .../cassandra/service/StorageService.java   | 28 +++-
 .../apache/cassandra/db/SystemKeyspaceTest.java |  6 -
 .../cassandra/gms/FailureDetectorTest.java  |  2 +-
 .../service/LeaveAndBootstrapTest.java  | 10 +--
 6 files changed, 44 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a0a61ac..d8b22f0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)
  * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/src/java/org/apache/cassandra/db/SystemKeyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/SystemKeyspace.java 
b/src/java/org/apache/cassandra/db/SystemKeyspace.java
index cc21435..7ce74a1 100644
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@ -23,11 +23,13 @@ import java.io.IOException;
 import java.net.InetAddress;
 import java.nio.ByteBuffer;
 import java.util.*;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import java.util.stream.StreamSupport;
 import javax.management.openmbean.OpenDataException;
 import javax.management.openmbean.TabularData;
+import java.util.concurrent.Future;
 
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableSet;
@@ -36,6 +38,10 @@ import com.google.common.io.ByteStreams;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.util.concurrent.Futures;
+
+import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.QueryProcessor;
@@ -687,29 +693,29 @@ public final class SystemKeyspace
 /**
  * Record tokens being used by another node
  */
-public static synchronized void updateTokens(InetAddress ep, 
Collection tokens)
+public static Future updateTokens(final InetAddress ep, final 
Collection tokens, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return Futures.immediateFuture(null);
 
 String req = "INSERT INTO system.%s (peer, tokens) VALUES (?, ?)";
-executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens));
+return executorService.submit((Runnable) () -> 
executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens)));
 }
 
-public static synchronized void updatePreferredIP(InetAddress ep, 
InetAddress preferred_ip)
+public static void updatePreferredIP(InetAddress ep, InetAddress 
preferred_ip)
 {
 String req = "INSERT INTO system.%s (peer, preferred_ip) VALUES (?, 
?)";
 executeInternal(String.format(req, PEERS), ep, preferred_ip);
 forceBlockingFlush(PEERS);
 }
 
-public static synchronized void updatePeerInfo(InetAddress ep, String 
columnName, Object value)
+public static Future updatePeerInfo(final InetAddress ep, final String 
columnName, final Object value, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return Futures.immediateFuture(null);
 
 String req = "INSERT INTO system.%s (peer, %s) VALUES (?, ?)";
-executeInternal(String.format(req, PEERS, columnName), ep, value);
+return 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/97b1c3ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/97b1c3ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/97b1c3ce

Branch: refs/heads/trunk
Commit: 97b1c3ce8274bccbf106e418fc5fcb0df364811c
Parents: 4803742 7f8417c
Author: Jason Brown 
Authored: Mon Aug 21 15:54:35 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:54:35 2017 -0700

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Gossip thread slows down when using batch commit log

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 dc32ed80d -> ec85b4a96
  refs/heads/cassandra-3.11 b726f26ae -> 7f8417c40
  refs/heads/trunk 4803742ec -> 97b1c3ce8


Gossip thread slows down when using batch commit log

patch by jasobrown; reviwed by spodkowinski fot CASSANDRA-12966


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ec85b4a9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ec85b4a9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ec85b4a9

Branch: refs/heads/cassandra-3.0
Commit: ec85b4a96390ef5bf85acac42f9c93a68620b668
Parents: dc32ed8
Author: Jason Brown 
Authored: Mon Nov 28 15:22:14 2016 -0800
Committer: Jason Brown 
Committed: Mon Aug 21 15:48:09 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/SystemKeyspace.java | 22 +--
 .../cassandra/service/StorageService.java   | 28 +++-
 .../apache/cassandra/db/SystemKeyspaceTest.java |  6 -
 .../cassandra/gms/FailureDetectorTest.java  |  2 +-
 .../service/LeaveAndBootstrapTest.java  | 10 +--
 6 files changed, 44 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a0a61ac..d8b22f0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
  * Fix digest calculation for counter cells (CASSANDRA-13750)
  * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec85b4a9/src/java/org/apache/cassandra/db/SystemKeyspace.java
--
diff --git a/src/java/org/apache/cassandra/db/SystemKeyspace.java 
b/src/java/org/apache/cassandra/db/SystemKeyspace.java
index cc21435..7ce74a1 100644
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@ -23,11 +23,13 @@ import java.io.IOException;
 import java.net.InetAddress;
 import java.nio.ByteBuffer;
 import java.util.*;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import java.util.stream.StreamSupport;
 import javax.management.openmbean.OpenDataException;
 import javax.management.openmbean.TabularData;
+import java.util.concurrent.Future;
 
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableSet;
@@ -36,6 +38,10 @@ import com.google.common.io.ByteStreams;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.util.concurrent.Futures;
+
+import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.QueryProcessor;
@@ -687,29 +693,29 @@ public final class SystemKeyspace
 /**
  * Record tokens being used by another node
  */
-public static synchronized void updateTokens(InetAddress ep, 
Collection tokens)
+public static Future updateTokens(final InetAddress ep, final 
Collection tokens, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return Futures.immediateFuture(null);
 
 String req = "INSERT INTO system.%s (peer, tokens) VALUES (?, ?)";
-executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens));
+return executorService.submit((Runnable) () -> 
executeInternal(String.format(req, PEERS), ep, tokensAsSet(tokens)));
 }
 
-public static synchronized void updatePreferredIP(InetAddress ep, 
InetAddress preferred_ip)
+public static void updatePreferredIP(InetAddress ep, InetAddress 
preferred_ip)
 {
 String req = "INSERT INTO system.%s (peer, preferred_ip) VALUES (?, 
?)";
 executeInternal(String.format(req, PEERS), ep, preferred_ip);
 forceBlockingFlush(PEERS);
 }
 
-public static synchronized void updatePeerInfo(InetAddress ep, String 
columnName, Object value)
+public static Future updatePeerInfo(final InetAddress ep, final String 
columnName, final Object value, ExecutorService executorService)
 {
 if (ep.equals(FBUtilities.getBroadcastAddress()))
-return;
+return 

[jira] [Updated] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13649:

Resolution: Fixed
  Reviewer: Jason Brown
Status: Resolved  (was: Patch Available)

> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
>  Labels: patch
> Attachments: 
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch, 
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
>  Just want to make sure that we don't have to change anything related to the 
> exception handling in our pipeline and that this isn't a netty issue. 
> Actually if this causes flakiness but is otherwise harmless, we should do 
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() 
> failed}} error also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135964#comment-16135964
 ] 

Jason Brown commented on CASSANDRA-13649:
-

I took the editorial liberty of committing to 2.2 and up. sha is 
{{e1aa7d32c95ff3f06de97803a186ff432237ecab}}

Thanks for the the patch, [~norman]!

> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
>  Labels: patch
> Attachments: 
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch, 
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
>  Just want to make sure that we don't have to change anything related to the 
> exception handling in our pipeline and that this isn't a netty issue. 
> Actually if this causes flakiness but is otherwise harmless, we should do 
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() 
> failed}} error also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc32ed80
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc32ed80
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc32ed80

Branch: refs/heads/cassandra-3.0
Commit: dc32ed80d727ca644d2162fbf0fb316811cd1adf
Parents: 8378bfc e1aa7d3
Author: Jason Brown 
Authored: Mon Aug 21 15:33:19 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:36:38 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/CHANGES.txt
--
diff --cc CHANGES.txt
index 6faaa48,2fbb7e9..a0a61ac
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,26 -1,6 +1,27 @@@
 -2.2.11
 +3.0.15
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Message.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Server.java
--
diff --cc src/java/org/apache/cassandra/transport/Server.java
index f35d507,c91d37d..7df194d
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@@ -327,10 -304,15 +328,18 @@@ public class Server implements Cassandr
  pipeline.addLast("messageDecoder", messageDecoder);
  pipeline.addLast("messageEncoder", messageEncoder);
  
+ // The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+ // on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+ // eventExecutorGroup may not enforce strict ordering for channel 
events.
+ // As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+ // correctly handled before the handler itself is removed.
+ // See https://issues.apache.org/jira/browse/CASSANDRA-13649
+ pipeline.addLast("exceptionHandler", exceptionHandler);
+ 
 -pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 +if 

[01/10] cassandra git commit: Uncaught exceptions in Netty pipeline

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 9b6fd54a4 -> e1aa7d32c
  refs/heads/cassandra-3.0 8378bfc51 -> dc32ed80d
  refs/heads/cassandra-3.11 1619413e5 -> b726f26ae
  refs/heads/trunk 6e42dd215 -> 4803742ec


Uncaught exceptions in Netty pipeline

patch by norman maurer; reviewed by jasobrown for CASSANDRA-13649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e1aa7d32
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e1aa7d32
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e1aa7d32

Branch: refs/heads/cassandra-2.2
Commit: e1aa7d32c95ff3f06de97803a186ff432237ecab
Parents: 9b6fd54
Author: Norman Maurer 
Authored: Fri Aug 18 10:32:39 2017 +0200
Committer: Jason Brown 
Committed: Mon Aug 21 15:31:47 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5c1d1e5..2fbb7e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Message.java
--
diff --git a/src/java/org/apache/cassandra/transport/Message.java 
b/src/java/org/apache/cassandra/transport/Message.java
index e4f5cac..2c2048f 100644
--- a/src/java/org/apache/cassandra/transport/Message.java
+++ b/src/java/org/apache/cassandra/transport/Message.java
@@ -540,10 +540,14 @@ public abstract class Message
 flusher.queued.add(item);
 flusher.start();
 }
+}
+
+@ChannelHandler.Sharable
+public static final class ExceptionHandler extends 
ChannelInboundHandlerAdapter
+{
 
 @Override
 public void exceptionCaught(final ChannelHandlerContext ctx, Throwable 
cause)
-throws Exception
 {
 // Provide error message to client in case channel is still open
 UnexpectedChannelExceptionHandler handler = new 
UnexpectedChannelExceptionHandler(ctx.channel(), false);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Server.java
--
diff --git a/src/java/org/apache/cassandra/transport/Server.java 
b/src/java/org/apache/cassandra/transport/Server.java
index d1047f9..c91d37d 100644
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@ -270,6 +270,7 @@ public class Server implements CassandraDaemon.Server
 private static final Frame.Decompressor frameDecompressor = new 
Frame.Decompressor();
 private static final Frame.Compressor frameCompressor = new 
Frame.Compressor();
 private static final Frame.Encoder frameEncoder = new Frame.Encoder();
+private static final Message.ExceptionHandler exceptionHandler = new 
Message.ExceptionHandler();
 private static final Message.Dispatcher dispatcher = new 
Message.Dispatcher();
 private static final ConnectionLimitHandler connectionLimitHandler = 
new ConnectionLimitHandler();
 
@@ -303,6 +304,14 @@ public class Server implements CassandraDaemon.Server
 pipeline.addLast("messageDecoder", messageDecoder);
 pipeline.addLast("messageEncoder", messageEncoder);
 
+// The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+// on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+// eventExecutorGroup may not enforce strict ordering for channel 
events.
+// As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+// correctly handled before the handler itself is removed.
+// See https://issues.apache.org/jira/browse/CASSANDRA-13649
+pipeline.addLast("exceptionHandler", exceptionHandler);
+
 pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 }
 }

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b726f26a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b726f26a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b726f26a

Branch: refs/heads/cassandra-3.11
Commit: b726f26aee0aa0e42a4d96a69fa8a761610c928c
Parents: 1619413 dc32ed8
Author: Jason Brown 
Authored: Mon Aug 21 15:36:58 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:37:43 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/src/java/org/apache/cassandra/transport/Message.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/src/java/org/apache/cassandra/transport/Server.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc32ed80
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc32ed80
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc32ed80

Branch: refs/heads/trunk
Commit: dc32ed80d727ca644d2162fbf0fb316811cd1adf
Parents: 8378bfc e1aa7d3
Author: Jason Brown 
Authored: Mon Aug 21 15:33:19 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:36:38 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/CHANGES.txt
--
diff --cc CHANGES.txt
index 6faaa48,2fbb7e9..a0a61ac
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,26 -1,6 +1,27 @@@
 -2.2.11
 +3.0.15
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Message.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Server.java
--
diff --cc src/java/org/apache/cassandra/transport/Server.java
index f35d507,c91d37d..7df194d
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@@ -327,10 -304,15 +328,18 @@@ public class Server implements Cassandr
  pipeline.addLast("messageDecoder", messageDecoder);
  pipeline.addLast("messageEncoder", messageEncoder);
  
+ // The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+ // on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+ // eventExecutorGroup may not enforce strict ordering for channel 
events.
+ // As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+ // correctly handled before the handler itself is removed.
+ // See https://issues.apache.org/jira/browse/CASSANDRA-13649
+ pipeline.addLast("exceptionHandler", exceptionHandler);
+ 
 -pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 +if 

[02/10] cassandra git commit: Uncaught exceptions in Netty pipeline

[jasobrown]

Uncaught exceptions in Netty pipeline

patch by norman maurer; reviewed by jasobrown for CASSANDRA-13649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e1aa7d32
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e1aa7d32
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e1aa7d32

Branch: refs/heads/cassandra-3.0
Commit: e1aa7d32c95ff3f06de97803a186ff432237ecab
Parents: 9b6fd54
Author: Norman Maurer 
Authored: Fri Aug 18 10:32:39 2017 +0200
Committer: Jason Brown 
Committed: Mon Aug 21 15:31:47 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5c1d1e5..2fbb7e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Message.java
--
diff --git a/src/java/org/apache/cassandra/transport/Message.java 
b/src/java/org/apache/cassandra/transport/Message.java
index e4f5cac..2c2048f 100644
--- a/src/java/org/apache/cassandra/transport/Message.java
+++ b/src/java/org/apache/cassandra/transport/Message.java
@@ -540,10 +540,14 @@ public abstract class Message
 flusher.queued.add(item);
 flusher.start();
 }
+}
+
+@ChannelHandler.Sharable
+public static final class ExceptionHandler extends 
ChannelInboundHandlerAdapter
+{
 
 @Override
 public void exceptionCaught(final ChannelHandlerContext ctx, Throwable 
cause)
-throws Exception
 {
 // Provide error message to client in case channel is still open
 UnexpectedChannelExceptionHandler handler = new 
UnexpectedChannelExceptionHandler(ctx.channel(), false);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Server.java
--
diff --git a/src/java/org/apache/cassandra/transport/Server.java 
b/src/java/org/apache/cassandra/transport/Server.java
index d1047f9..c91d37d 100644
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@ -270,6 +270,7 @@ public class Server implements CassandraDaemon.Server
 private static final Frame.Decompressor frameDecompressor = new 
Frame.Decompressor();
 private static final Frame.Compressor frameCompressor = new 
Frame.Compressor();
 private static final Frame.Encoder frameEncoder = new Frame.Encoder();
+private static final Message.ExceptionHandler exceptionHandler = new 
Message.ExceptionHandler();
 private static final Message.Dispatcher dispatcher = new 
Message.Dispatcher();
 private static final ConnectionLimitHandler connectionLimitHandler = 
new ConnectionLimitHandler();
 
@@ -303,6 +304,14 @@ public class Server implements CassandraDaemon.Server
 pipeline.addLast("messageDecoder", messageDecoder);
 pipeline.addLast("messageEncoder", messageEncoder);
 
+// The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+// on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+// eventExecutorGroup may not enforce strict ordering for channel 
events.
+// As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+// correctly handled before the handler itself is removed.
+// See https://issues.apache.org/jira/browse/CASSANDRA-13649
+pipeline.addLast("exceptionHandler", exceptionHandler);
+
 pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b726f26a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b726f26a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b726f26a

Branch: refs/heads/trunk
Commit: b726f26aee0aa0e42a4d96a69fa8a761610c928c
Parents: 1619413 dc32ed8
Author: Jason Brown 
Authored: Mon Aug 21 15:36:58 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:37:43 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/src/java/org/apache/cassandra/transport/Message.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b726f26a/src/java/org/apache/cassandra/transport/Server.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4803742e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4803742e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4803742e

Branch: refs/heads/trunk
Commit: 4803742eca5349994e2fb7118008c9b30b9dac4e
Parents: 6e42dd2 b726f26
Author: Jason Brown 
Authored: Mon Aug 21 15:39:06 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:39:40 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4803742e/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4803742e/src/java/org/apache/cassandra/transport/Server.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[04/10] cassandra git commit: Uncaught exceptions in Netty pipeline

[jasobrown]

Uncaught exceptions in Netty pipeline

patch by norman maurer; reviewed by jasobrown for CASSANDRA-13649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e1aa7d32
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e1aa7d32
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e1aa7d32

Branch: refs/heads/trunk
Commit: e1aa7d32c95ff3f06de97803a186ff432237ecab
Parents: 9b6fd54
Author: Norman Maurer 
Authored: Fri Aug 18 10:32:39 2017 +0200
Committer: Jason Brown 
Committed: Mon Aug 21 15:31:47 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5c1d1e5..2fbb7e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Message.java
--
diff --git a/src/java/org/apache/cassandra/transport/Message.java 
b/src/java/org/apache/cassandra/transport/Message.java
index e4f5cac..2c2048f 100644
--- a/src/java/org/apache/cassandra/transport/Message.java
+++ b/src/java/org/apache/cassandra/transport/Message.java
@@ -540,10 +540,14 @@ public abstract class Message
 flusher.queued.add(item);
 flusher.start();
 }
+}
+
+@ChannelHandler.Sharable
+public static final class ExceptionHandler extends 
ChannelInboundHandlerAdapter
+{
 
 @Override
 public void exceptionCaught(final ChannelHandlerContext ctx, Throwable 
cause)
-throws Exception
 {
 // Provide error message to client in case channel is still open
 UnexpectedChannelExceptionHandler handler = new 
UnexpectedChannelExceptionHandler(ctx.channel(), false);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Server.java
--
diff --git a/src/java/org/apache/cassandra/transport/Server.java 
b/src/java/org/apache/cassandra/transport/Server.java
index d1047f9..c91d37d 100644
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@ -270,6 +270,7 @@ public class Server implements CassandraDaemon.Server
 private static final Frame.Decompressor frameDecompressor = new 
Frame.Decompressor();
 private static final Frame.Compressor frameCompressor = new 
Frame.Compressor();
 private static final Frame.Encoder frameEncoder = new Frame.Encoder();
+private static final Message.ExceptionHandler exceptionHandler = new 
Message.ExceptionHandler();
 private static final Message.Dispatcher dispatcher = new 
Message.Dispatcher();
 private static final ConnectionLimitHandler connectionLimitHandler = 
new ConnectionLimitHandler();
 
@@ -303,6 +304,14 @@ public class Server implements CassandraDaemon.Server
 pipeline.addLast("messageDecoder", messageDecoder);
 pipeline.addLast("messageEncoder", messageEncoder);
 
+// The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+// on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+// eventExecutorGroup may not enforce strict ordering for channel 
events.
+// As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+// correctly handled before the handler itself is removed.
+// See https://issues.apache.org/jira/browse/CASSANDRA-13649
+pipeline.addLast("exceptionHandler", exceptionHandler);
+
 pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[03/10] cassandra git commit: Uncaught exceptions in Netty pipeline

[jasobrown]

Uncaught exceptions in Netty pipeline

patch by norman maurer; reviewed by jasobrown for CASSANDRA-13649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e1aa7d32
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e1aa7d32
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e1aa7d32

Branch: refs/heads/cassandra-3.11
Commit: e1aa7d32c95ff3f06de97803a186ff432237ecab
Parents: 9b6fd54
Author: Norman Maurer 
Authored: Fri Aug 18 10:32:39 2017 +0200
Committer: Jason Brown 
Committed: Mon Aug 21 15:31:47 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5c1d1e5..2fbb7e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Message.java
--
diff --git a/src/java/org/apache/cassandra/transport/Message.java 
b/src/java/org/apache/cassandra/transport/Message.java
index e4f5cac..2c2048f 100644
--- a/src/java/org/apache/cassandra/transport/Message.java
+++ b/src/java/org/apache/cassandra/transport/Message.java
@@ -540,10 +540,14 @@ public abstract class Message
 flusher.queued.add(item);
 flusher.start();
 }
+}
+
+@ChannelHandler.Sharable
+public static final class ExceptionHandler extends 
ChannelInboundHandlerAdapter
+{
 
 @Override
 public void exceptionCaught(final ChannelHandlerContext ctx, Throwable 
cause)
-throws Exception
 {
 // Provide error message to client in case channel is still open
 UnexpectedChannelExceptionHandler handler = new 
UnexpectedChannelExceptionHandler(ctx.channel(), false);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e1aa7d32/src/java/org/apache/cassandra/transport/Server.java
--
diff --git a/src/java/org/apache/cassandra/transport/Server.java 
b/src/java/org/apache/cassandra/transport/Server.java
index d1047f9..c91d37d 100644
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@ -270,6 +270,7 @@ public class Server implements CassandraDaemon.Server
 private static final Frame.Decompressor frameDecompressor = new 
Frame.Decompressor();
 private static final Frame.Compressor frameCompressor = new 
Frame.Compressor();
 private static final Frame.Encoder frameEncoder = new Frame.Encoder();
+private static final Message.ExceptionHandler exceptionHandler = new 
Message.ExceptionHandler();
 private static final Message.Dispatcher dispatcher = new 
Message.Dispatcher();
 private static final ConnectionLimitHandler connectionLimitHandler = 
new ConnectionLimitHandler();
 
@@ -303,6 +304,14 @@ public class Server implements CassandraDaemon.Server
 pipeline.addLast("messageDecoder", messageDecoder);
 pipeline.addLast("messageEncoder", messageEncoder);
 
+// The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+// on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+// eventExecutorGroup may not enforce strict ordering for channel 
events.
+// As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+// correctly handled before the handler itself is removed.
+// See https://issues.apache.org/jira/browse/CASSANDRA-13649
+pipeline.addLast("exceptionHandler", exceptionHandler);
+
 pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc32ed80
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc32ed80
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc32ed80

Branch: refs/heads/cassandra-3.11
Commit: dc32ed80d727ca644d2162fbf0fb316811cd1adf
Parents: 8378bfc e1aa7d3
Author: Jason Brown 
Authored: Mon Aug 21 15:33:19 2017 -0700
Committer: Jason Brown 
Committed: Mon Aug 21 15:36:38 2017 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/transport/Message.java | 6 +-
 src/java/org/apache/cassandra/transport/Server.java  | 9 +
 3 files changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/CHANGES.txt
--
diff --cc CHANGES.txt
index 6faaa48,2fbb7e9..a0a61ac
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,26 -1,6 +1,27 @@@
 -2.2.11
 +3.0.15
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Message.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc32ed80/src/java/org/apache/cassandra/transport/Server.java
--
diff --cc src/java/org/apache/cassandra/transport/Server.java
index f35d507,c91d37d..7df194d
--- a/src/java/org/apache/cassandra/transport/Server.java
+++ b/src/java/org/apache/cassandra/transport/Server.java
@@@ -327,10 -304,15 +328,18 @@@ public class Server implements Cassandr
  pipeline.addLast("messageDecoder", messageDecoder);
  pipeline.addLast("messageEncoder", messageEncoder);
  
+ // The exceptionHandler will take care of handling 
exceptionCaught(...) events while still running
+ // on the same EventLoop as all previous added handlers in the 
pipeline. This is important as the used
+ // eventExecutorGroup may not enforce strict ordering for channel 
events.
+ // As the exceptionHandler runs in the EventLoop as the previous 
handlers we are sure all exceptions are
+ // correctly handled before the handler itself is removed.
+ // See https://issues.apache.org/jira/browse/CASSANDRA-13649
+ pipeline.addLast("exceptionHandler", exceptionHandler);
+ 
 -pipeline.addLast(server.eventExecutorGroup, "executor", 
dispatcher);
 +if 

[jira] [Updated] (CASSANDRA-13655) Range deletes in a CAS batch are ignored

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-13655:

Reviewer: Sylvain Lebresne

> Range deletes in a CAS batch are ignored
> 
>
> Key: CASSANDRA-13655
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13655
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Blocker
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Range deletes in a CAS batch are ignored 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135814#comment-16135814
 ] 

Aleksey Yeschenko commented on CASSANDRA-13740:
---

Hey. There are a few [code style|https://wiki.apache.org/cassandra/CodeStyle] 
issues: we don't use {{final}} for arguments and local variables, brackets go 
to new lines always. And the patch doesn't wait for {{closeWriter}} future to 
be completed.

And a more interesting issue is that of the delay. {{RING_DELAY}} doesn't have 
anything to do with hints. What does is write timeouts, and 
{{MessagingService}} 's timeout reporter the callbacks expiring map firing - 
that's where the race ultimately is.

Also, we aren't fixing the issue of {{nodetool truncatehints}} not being able 
to clean up after we excise.

The more I think about it, the more I'm inclined to just correct that last 
issue and leave everything else be as is (and also commit your unit tests, 
thanks for those).

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[sankalp kohli (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135790#comment-16135790
 ] 

sankalp kohli commented on CASSANDRA-8457:
--

[~slebresne]  I know you and Jason chatted on IRC. Are you +1 on 
this...Streaming patch has +1 from Ariel 

> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.x
>
> Attachments: 8457-load.tgz
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135748#comment-16135748
 ] 

mck commented on CASSANDRA-13418:
-

{quote}And also as the parent function already use a mutable set…{quote}
fair enough, i was too lazy to check so far :-)

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13756) StreamingHistogram is not thread safe

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135664#comment-16135664
 ] 

Hannu Kröger commented on CASSANDRA-13756:
--

[~jjirsa] Serialization probably needs to be synchronized somehow as well. 
Seems like sstable statistics can be corrupted at the moment because of this 
issue and although I am not sure it looks like that this patch probably doesn't 
address.

> StreamingHistogram is not thread safe
> -
>
> Key: CASSANDRA-13756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13756
> Project: Cassandra
>  Issue Type: Bug
>Reporter: xiangzhou xia
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x
>
>
> When we test C*3 in shadow cluster, we notice after a period of time, several 
> data node suddenly run into 100% cpu and stop process query anymore.
> After investigation, we found that threads are stuck on the sum() in 
> streaminghistogram class. Those are jmx threads that working on expose 
> getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is 
> a chance that multiple jmx thread is access streaminghistogram at the same 
> time).  
> After further investigation, we find that the optimization in CASSANDRA-13038 
> led to a spool flush every time when we call sum(). Since TreeMap is not 
> thread safe, threads will be stuck when multiple threads visit sum() at the 
> same time.
> There are two approaches to solve this issue. 
> The first one is to add a lock to the flush in sum() which will introduce 
> some extra overhead to streaminghistogram.
> The second one is to avoid streaminghistogram to be access by multiple 
> threads. For our specific case, is to remove the metrics we added.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13756) StreamingHistogram is not thread safe

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135593#comment-16135593
 ] 

Hannu Kröger commented on CASSANDRA-13756:
--

Linking to SSTable Corruption ticket which this same bug seems to cause.

> StreamingHistogram is not thread safe
> -
>
> Key: CASSANDRA-13756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13756
> Project: Cassandra
>  Issue Type: Bug
>Reporter: xiangzhou xia
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x
>
>
> When we test C*3 in shadow cluster, we notice after a period of time, several 
> data node suddenly run into 100% cpu and stop process query anymore.
> After investigation, we found that threads are stuck on the sum() in 
> streaminghistogram class. Those are jmx threads that working on expose 
> getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is 
> a chance that multiple jmx thread is access streaminghistogram at the same 
> time).  
> After further investigation, we find that the optimization in CASSANDRA-13038 
> led to a spool flush every time when we call sum(). Since TreeMap is not 
> thread safe, threads will be stuck when multiple threads visit sum() at the 
> same time.
> There are two approaches to solve this issue. 
> The first one is to add a lock to the flush in sum() which will introduce 
> some extra overhead to streaminghistogram.
> The second one is to avoid streaminghistogram to be access by multiple 
> threads. For our specific case, is to remove the metrics we added.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135576#comment-16135576
 ] 

Hannu Kröger commented on CASSANDRA-13752:
--

I think I got it confirmed. StreamingHistogram size stored had actually value 
that was less than the amount of elements serialized. It was off by one.

Only thing I am worried is that if CASSANDRA-13756 patch fixes also the 
serialization issue.

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Priority: Blocker
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13782) Cassandra RPM has wrong owner for /usr/share directories

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannu Kröger updated CASSANDRA-13782:
-
Summary: Cassandra RPM has wrong owner for /usr/share directories  (was: 
Cassandra RPM has wrong owner for directories)

> Cassandra RPM has wrong owner for /usr/share directories
> 
>
> Key: CASSANDRA-13782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13782
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Hannu Kröger
>  Labels: lhf
>
> Some Cassandra RPM directories are owned by cassandra user against the fedora 
> package guidelines.
> Offending lines: 
> https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136
> "Permissions on files MUST be set properly. Inside of /usr, files should be 
> owned by root:root unless a more specific user or group is needed for 
> security."
> - 
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13782) Cassandra RPM has wrong owner for directories

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannu Kröger updated CASSANDRA-13782:
-
Labels: lhf  (was: )

> Cassandra RPM has wrong owner for directories
> -
>
> Key: CASSANDRA-13782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13782
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Hannu Kröger
>  Labels: lhf
>
> Some Cassandra RPM directories are owned by cassandra user against the fedora 
> package guidelines.
> Offending lines: 
> https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136
> "Permissions on files MUST be set properly. Inside of /usr, files should be 
> owned by root:root unless a more specific user or group is needed for 
> security."
> - 
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13782) Cassandra RPM has wrong owner for directories

[JIRA]

Hannu Kröger created CASSANDRA-13782:


 Summary: Cassandra RPM has wrong owner for directories
 Key: CASSANDRA-13782
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13782
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Reporter: Hannu Kröger


Some Cassandra RPM directories are owned by cassandra user against the fedora 
package guidelines.

Offending lines: 
https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136

"Permissions on files MUST be set properly. Inside of /usr, files should be 
owned by root:root unless a more specific user or group is needed for security."
- 
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135453#comment-16135453
 ] 

Hannu Kröger commented on CASSANDRA-13752:
--

I debugged the code by loading Cassandra in a debugger and opening that sstable 
(with empty data & index) and after deserializing StreamingHistogram the data 
seems to be garbage.

Could this be a side effect of this bug: 
https://issues.apache.org/jira/browse/CASSANDRA-13756 ?

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Priority: Blocker
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13781) Cassandra 3.10 jvm.options default -XX:+UseNUMA may core OpenJDK on some aarch64 machines at os::Linux::libnuma_init()

[Yangzheng Bai (JIRA)]

Yangzheng Bai created CASSANDRA-13781:
-

 Summary: Cassandra 3.10 jvm.options default -XX:+UseNUMA may core 
OpenJDK on some aarch64 machines at os::Linux::libnuma_init()
 Key: CASSANDRA-13781
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13781
 Project: Cassandra
  Issue Type: Bug
Reporter: Yangzheng Bai


We tried Oracle JDK, Linaro JDK, Zulu JDK and default OpenJDK, all of them 
cored at the same place: os::Linux::libnuma_init() -> 
os::Linux::rebuild_cpu_to_node_map()

Removed -XX:+UseNUMA option solved the problem. Therefore we should remove 
-XX:+UseNUMA as an default option on aarch64 machines because x86_64 and 
aarch64 may have different implementations on how NUMA is reported. And 
-XX:+UseNUMA is currently not supporting G1GC anyway 
(https://bugs.openjdk.java.net/browse/JDK-8046147), adding it as default option 
may not have performance gain for future Java default G1GC.

{noformat}
root@1pqd-1:/usr/share/cassandra/lib# /usr/sbin/cassandra -R -f
*** Error in `java': free(): invalid pointer: 0x800058c0 ***
Aborted (core dumped)

(gdb) bt
#0  0x8719d528 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x8719e9e0 in __GI_abort () at abort.c:89
#2  0x871d48c4 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x87285550 "*** Error in `%s': %s: 0x%s ***\n") at 
../sysdeps/posix/libc_fatal.c:175
#3  0x871da2ec in malloc_printerr (action=, 
str=0x87285720 "free(): invalid pointer",
ptr=, ar_ptr=) at malloc.c:5006
#4  0x871db088 in _int_free (av=0x872ad9a0 , 
p=, have_lock=0) at malloc.c:3867
#5  0x86dbccc4 in os::Linux::rebuild_cpu_to_node_map() ()
   from /usr/lib/jvm/java-8-linaro/jre/lib/aarch64/server/libjvm.so
#6  0x86dbd048 in os::Linux::libnuma_init() () from 
/usr/lib/jvm/java-8-linaro/jre/lib/aarch64/server/libjvm.so
#7  0x86dc34ac in os::init_2() () from 
/usr/lib/jvm/java-8-linaro/jre/lib/aarch64/server/libjvm.so
#8  0x86f0f374 in Threads::create_vm(JavaVMInitArgs*, bool*) ()
   from /usr/lib/jvm/java-8-linaro/jre/lib/aarch64/server/libjvm.so
#9  0x86bb7b80 in JNI_CreateJavaVM () from 
/usr/lib/jvm/java-8-linaro/jre/lib/aarch64/server/libjvm.so
#10 0x872ba75c in InitializeJVM (ifn=, 
penv=0x86533a38, pvm=0x86533a30)
at 
/home/buildslave/workspace/jdk8-build-release/BUILD_TYPE/release/JVM_VARIANT/server/label/aarch64-06/jdk8u-server-release-1704/jdk/src/share/bin/java.c:1215
#11 JavaMain (_args=)
at 
/home/buildslave/workspace/jdk8-build-release/BUILD_TYPE/release/JVM_VARIANT/server/label/aarch64-06/jdk8u-server-release-1704/jdk/src/share/bin/java.c:376
#12 0x87133fc4 in start_thread (arg=0x872ba6d0 ) at 
pthread_create.c:335
#13 0x87232290 in thread_start () at 
../sysdeps/unix/sysv/linux/aarch64/clone.S:89
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13780) ADD Node streaming throughput performance

[Kevin Rivait (JIRA)]

Kevin Rivait created CASSANDRA-13780:


 Summary: ADD Node streaming throughput performance
 Key: CASSANDRA-13780
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux

Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):40
On-line CPU(s) list:   0-39
Thread(s) per core:2
Core(s) per socket:10
Socket(s): 2
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 79
Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Stepping:  1
CPU MHz:   2199.869
BogoMIPS:  4399.36
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  25600K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39

 total   used   free sharedbuffers cached
Mem:  252G   217G34G   708K   308M   149G
-/+ buffers/cache:67G   185G
Swap:  16G 0B16G



Reporter: Kevin Rivait
Priority: Blocker
 Fix For: 3.0.9


Problem: Adding a new node to a large cluster runs at least 1000x slower than 
what the network and node hardware capacity can support, taking several days 
per new node.  Adjusting stream throughput and other YAML parameters seems to 
have no effect on performance.  Essentially, it appears that Cassandra has an 
architecture scalability growth problem when adding new nodes to a moderate to 
high data ingestion cluster because Cassandra cannot add new node capacity fast 
enough to keep up with increasing data ingestion volumes and growth.

Initial Configuration: 
Running 3.0.9 and have implemented TWCS on one of our largest table.
Largest table partitioned on (ID, MM)  using 1 day buckets with a TTL of 60 
days.
Next release will change partitioning to (ID, MMDD) so that partitions are 
aligned with daily TWCS buckets.
Each node is currently creating roughly a 30GB SSTable per day.
TWCS working as expected,  daily SSTables are dropping off daily after 70 days 
( 60 + 10 day grace)
Current deployment is a 28 node 2 datacenter cluster, 14 nodes in each DC , 
replication factor 3
Data directories are backed with 4 - 2TB SSDs on each node  and a 1 800GB SSD 
for commit logs.
Requirement is to double cluster size, capacity, and ingestion volume within a 
few weeks.

Observed Behavior:
1. streaming throughput during add node – we observed maximum 6 Mb/s streaming 
from each of the 14 nodes on a 20Gb/s switched network, taking at least 106 
hours for each node to join cluster and each node is only about 2.2 TB is size.

2. compaction on the newly added node - compaction has fallen behind, with 
anywhere from 4,000 to 10,000 SSTables at any given time.  It took 3 weeks for 
compaction to finish on each newly added node.   Increasing number of 
compaction threads to match number of CPU (40)  and increasing compaction 
throughput to 32MB/s seemed to be the sweet spot. 

3. TWCS buckets on new node, data streamed to this node over 4 1/2 days.  
Compaction correctly placed the data in daily files, but the problem is the 
file dates reflect when compaction created the file and not the date of the 
last record written in the TWCS bucket, which will cause the files to remain 
around much longer than necessary.  

Two Questions:
1. What can be done to substantially improve the performance of adding a new 
node?
2. Can compaction on TWCS partitions for newly added nodes change the file 
create date to match the highest date record in the file -or- add another piece 
of meta-data to the TWCS files that reflect the file drop date so that TWCS 
partitions can be dropped consistently?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135283#comment-16135283
 ] 

Romain GERARD edited comment on CASSANDRA-13418 at 8/21/17 3:22 PM:


Initial review comments: 
* CHANGES.txt needs a line -> OK
* I also think so as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.


was (Author: rgerard):
Initial review comments: 
* CHANGES.txt needs a line -> OK
* I also think so as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135283#comment-16135283
 ] 

Romain GERARD edited comment on CASSANDRA-13418 at 8/21/17 3:20 PM:


Initial review comments: 
* CHANGES.txt needs a line -> OK
* I also think so as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.


was (Author: rgerard):
Initial review comments: 
* CHANGES.txt needs a line -> OK
* I also think so as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135283#comment-16135283
 ] 

Romain GERARD edited comment on CASSANDRA-13418 at 8/21/17 3:20 PM:


Initial review comments: 
* CHANGES.txt needs a line -> OK
* I also think so as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.


was (Author: rgerard):
Initial review comments: 
* CHANGES.txt needs a line -> OK
* I think so also as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135283#comment-16135283
 ] 

Romain GERARD commented on CASSANDRA-13418:
---

Initial review comments: 
* CHANGES.txt needs a line -> OK
* I think so also as it greatly help having a stable behavior when using TWCS 
for time series
* Not at all, it was just to pack things together and to inform the reader that 
a TWCSCompactionController exist
* OK

Trivial stuff :
* Ok
* I don't like to return Immutable collections when only the base type (Set, 
List, Map,...) is specified as due to the type erasure someone will get burn at 
runtime with that (due to unchecked exception). And also as the parent function 
already use a mutable set I sticked with that because returning sometime a 
mutable set and sometime an immutable set is kind of a leaky abstraction for me 
(Will check if I can change everything for an ImmutableSet) 
* OK
* OK

Will propose an other patch tomorrow.

P.S: The patch has been running in production since last Friday without hickups.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13773) cassandra-stress writes even data when n=0

[Eduard Tudenhoefner (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135250#comment-16135250
 ] 

Eduard Tudenhoefner commented on CASSANDRA-13773:
-

Skipping the command entirely sgtm

> cassandra-stress writes even data when n=0
> --
>
> Key: CASSANDRA-13773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13773
> Project: Cassandra
>  Issue Type: Bug
>  Components: Stress
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
>Priority: Minor
> Fix For: 3.0.15
>
>
> This is very unintuitive as
> {code}
> cassandra-stress write n=0 -rate threads=1
> {code}
> will do inserts even with *n=0*. I guess most people won't ever run with 
> *n=0* but this is a nice shortcut for creating some schema without using 
> *cqlsh*
> This is happening because we're writing *50k* rows of warmup data as can be 
> seen below:
> {code}
> cqlsh> select count(*) from keyspace1.standard1 ;
>  count
> ---
>  5
> (1 rows)
> {code}
> We can avoid writing warmup data using 
> {code}
> cassandra-stress write n=0 no-warmup -rate threads=1
> {code}
> but I would still expect to have *0* rows written when specifying *n=0*.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13640) CQLSH error when using 'login' to switch users

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135234#comment-16135234
 ] 

ZhaoYang commented on CASSANDRA-13640:
--

The fix looks good. 3.11/trunk are also broken for the same reason, need to fix 
as well.

> CQLSH error when using 'login' to switch users
> --
>
> Key: CASSANDRA-13640
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13640
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
>Priority: Minor
> Fix For: 3.0.x
>
>
> Using {{PasswordAuthenticator}} and {{CassandraAuthorizer}}:
> {code}
> bin/cqlsh -u cassandra -p cassandra
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.0.14-SNAPSHOT | CQL spec 3.4.0 | Native protocol 
> v4]
> Use HELP for help.
> cassandra@cqlsh> create role super with superuser = true and password = 'p' 
> and login = true;
> cassandra@cqlsh> login super;
> Password:
> super@cqlsh> list roles;
> 'Row' object has no attribute 'values'
> {code}
> When we initialize the Shell, we configure certain settings on the session 
> object such as
> {code}
> self.session.default_timeout = request_timeout
> self.session.row_factory = ordered_dict_factory
> self.session.default_consistency_level = cassandra.ConsistencyLevel.ONE
> {code}
> However, once we perform a LOGIN cmd, which calls do_login(..), we create a 
> new cluster/session object but actually never set those settings on the new 
> session.
> It isn't failing on 3.x. 
> As a workaround, it is possible to logout and log back in and things work 
> correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13719) Potential AssertionError during ReadRepair of range tombstone and partition deletions

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135108#comment-16135108
 ] 

Sylvain Lebresne commented on CASSANDRA-13719:
--

Gentle ping that I've just rebased the branches above and this is still good 
for review. I've tried running CI with CircleCI and [it looks 
ok|https://circleci.com/gh/pcmanus/cassandra/3#tests/containers/0] (the build 
is marked failed, but afaict it's just the eclipse-warnings target that failed 
and from the logs, it's not even clear it's a genuine failure; all unit tests 
looks good however) though it's the first time I try it and I'm not sure if/how 
to run dtests yet. 

> Potential AssertionError during ReadRepair of range tombstone and partition 
> deletions
> -
>
> Key: CASSANDRA-13719
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13719
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.11.x
>
>
> When reconciling range tombstones for read repair in 
> {{DataResolver.RepairMergeListener.MergeListener}}, when we check if there is 
> ongoing deletion repair for a source, we don't look for partition level 
> deletions which throw off the logic and can throw an AssertionError.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135105#comment-16135105
 ] 

Aleksey Yeschenko commented on CASSANDRA-13649:
---

3.0 still accepts non-critical bug fixes. Are we treating 2.2 like 2.1 or like 
3.0 at this point? I honestly don't remember anymore.

> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
>  Labels: patch
> Attachments: 
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch, 
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
>  Just want to make sure that we don't have to change anything related to the 
> exception handling in our pipeline and that this isn't a netty issue. 
> Actually if this causes flakiness but is otherwise harmless, we should do 
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() 
> failed}} error also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko reassigned CASSANDRA-13363:
-

Assignee: zhaoyan  (was: Aleksey Yeschenko)

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13363:
--
Status: Patch Available  (was: Open)

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko reassigned CASSANDRA-13363:
-

Assignee: Aleksey Yeschenko  (was: zhaoyan)

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13363:
--
Reviewer: Sam Tunnicliffe  (was: Aleksey Yeschenko)

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135088#comment-16135088
 ] 

Aleksey Yeschenko commented on CASSANDRA-13363:
---

[~zhaoyan] Thanks again for the initial analysis. And that patch would 100% 
prevent the AIOOBE, just not fix the underlying issue entirely.

A more comprehensive refactoring to fix the race for good: 
[3.0|https://github.com/iamaleksey/cassandra/tree/13363-3.0], 
[3.11|https://github.com/iamaleksey/cassandra/tree/13363-3.11], 
[4.0|https://github.com/iamaleksey/cassandra/tree/13363-4.0]. Not yet ready to 
commit, as waiting for tests results. [~beobal] can you please review?

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135083#comment-16135083
 ] 

Kurt Greaves commented on CASSANDRA-12783:
--

[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:trunk-11670?expand=1]

BatchlogManager is more or less done which is the main component. Haven't fixed 
{{LegacyBatchlogMigrator}} yet so plz ignore (the version there is from the 
last batchlog migration). Still got a bit of work to do on tests, but will 
continue with that after sorting out UUID issue. For interests sake, in the 
test {{testUUIDComparison}} id1 will occasionally return a UUID that is < id2, 
despite id2 having a timestamp (and uuid for that matter) that is far lower. 
This is a problem  
[here|https://github.com/apache/cassandra/compare/trunk...kgreav:trunk-11670?expand=1#diff-c77d5f1027ee5e8a49e106903a4ca937R318]
 the code will occasionally delete a batch that shouldn't be expired.

Also totally open to ideas/criticism regarding the whole design, specifically 
the expiry stuff as not sure yet if that will have other implications.

cc [~pauloricardomg] if you want to have a look

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135080#comment-16135080
 ] 

Benjamin Lerer commented on CASSANDRA-13006:


[~urandom] Sorry, this ticket felt out of my radar.

bq. IMO, the sanest strategy here would be to leave the creation of heap dumps 
to the JVM.

I fully agree with you. 
Initially, we started to catch {{OOM}} errors to prevent C* to run in an 
unknown state that could cause data corruption (see CASSANDRA-7507). The 
problem with that approach  was that Heap dumps were not created anymore on 
{{OOM}} errors. I tried to fix that in CASSANDRA-9861 but that approach seems 
to have some serious issues and limitations.

In April 2016, roughly at the same time that I was fixing CASSANDRA-9861, 
Oracle released the [JDK 
8u92|http://www.oracle.com/technetwork/java/javase/8u92-relnotes-2949471.html] 
which added 2 new JVM Options: {{ExitOnOutOfMemoryError}} and 
{{CrashOnOutOfMemoryError}}. 

With that in mind, my idea would be to:
# stop handling the {{OOM}} errors on the C* side (and producing Heap dump)
# add  {{ExitOnOutOfMemoryError}}  to the default JVM options in the startup 
scripts
# in the {{News.txt}} upgrade procedure: request people to use a java version 
>= {{8u92}} 

[~JoshuaMcKenzie] you worked on CASSANDRA-7507. Do you have any concern with 
the approach I am suggesting?

My only concern right now is that I know that some C* users are using Zing and 
I do not if it support the {{ExitOnOutOfMemoryError}} option.
I asked the Zing support and will update the ticket as soon as I know more.

 

> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by 
> default if the process encountered an OOM error. These heap dumps are stored 
> in the Apache Cassandra home directory unless configured otherwise (see 
> [Cassandra Support 
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
>  for this feature).
>  
> The creation and storage of heap dumps aides debugging and investigative 
> workflows, but is not be desirable for a production environment where these 
> heap dumps may occupy a large amount of disk space and require manual 
> intervention for cleanups. 
>  
> Managing heap dumps on out of memory errors and configuring the paths for 
> these heap dumps are available as JVM options in JVM. The current behavior 
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. 
>  
> A patch can be proposed here that would make the heap dump on OOM error honor 
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate 
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM 
> option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13299) Potential OOMs and lock contention in write path streams

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134733#comment-16134733
 ] 

ZhaoYang edited comment on CASSANDRA-13299 at 8/21/17 11:54 AM:


[trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13299-trunk]
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-13299 ]

Changes:

1. Throttle by number of base unfiltered. default is 100. 
2. A pair of open/close range tombstone could have any number of unshadowed 
rows in between. In the patch, when reaching the limit of each batch, if there 
is an open range-tombstone-mark, it will generate a corresponding close marker 
for it. It's to avoid handling range-tombstone-mark separately from row which 
costs 1 more read-before-write for each pair of markers. This also help to 
reduce the impact of a large range tombstone.
3. Partition deletion is only applied on first mutation to avoid reading entire 
partition more than once.


Note:
One partition deletion or a range deletion could cause huge number of view rows 
to be removed, thus view mutation may fail to apply due to WTE or 
max_mutation_size, but it could be resolved separately in CASSANDRA-12783. 
Here, I only address the issue of holding entire partition into memory when 
repairing base with mv.


was (Author: jasonstack):
[trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13299-trunk]
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-13299 ]

Changes:

1. Throttle by number of base unfiltered. default is 100. 
2. A pair of open/close range tombstone could have any number of unshadowed 
rows in between. In the patch, when reaching the limit of each batch, if there 
is an open range-tombstone-mark, it will generate a corresponding close marker 
for it. 



Note:
One partition deletion or a range deletion could cause huge number of view rows 
to be removed, thus view mutation may fail to apply due to WTE or 
max_mutation_size, but it could be resolved separately in CASSANDRA-12783. 
Here, I only address the issue of holding entire partition into memory when 
repairing base with mv.

> Potential OOMs and lock contention in write path streams
> 
>
> Key: CASSANDRA-13299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13299
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Assignee: ZhaoYang
>
> I see a potential OOM, when a stream (e.g. repair) goes through the write 
> path as it is with MVs.
> StreamReceiveTask gets a bunch of SSTableReaders. These produce rowiterators 
> and they again produce mutations. So every partition creates a single 
> mutation, which in case of (very) big partitions can result in (very) big 
> mutations. Those are created on heap and stay there until they finished 
> processing.
> I don't think it is necessary to create a single mutation for each partition. 
> Why don't we implement a PartitionUpdateGeneratorIterator that takes a 
> UnfilteredRowIterator and a max size and spits out PartitionUpdates to be 
> used to create and apply mutations?
> The max size should be something like min(reasonable_absolute_max_size, 
> max_mutation_size, commitlog_segment_size / 2). reasonable_absolute_max_size 
> could be like 16M or sth.
> A mutation shouldn't be too large as it also affects MV partition locking. 
> The longer a MV partition is locked during a stream, the higher chances are 
> that WTE's occur during streams.
> I could also imagine that a max number of updates per mutation regardless of 
> size in bytes could make sense to avoid lock contention.
> Love to get feedback and suggestions, incl. naming suggestions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13299) Potential OOMs and lock contention in write path streams

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134733#comment-16134733
 ] 

ZhaoYang edited comment on CASSANDRA-13299 at 8/21/17 11:26 AM:


[trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13299-trunk]
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-13299 ]

Changes:

1. Throttle by number of base unfiltered. default is 100. 
2. A pair of open/close range tombstone could have any number of unshadowed 
rows in between. In the patch, when reaching the limit of each batch, if there 
is an open range-tombstone-mark, it will generate a corresponding close marker 
for it. 



Note:
One partition deletion or a range deletion could cause huge number of view rows 
to be removed, thus view mutation may fail to apply due to WTE or 
max_mutation_size, but it could be resolved separately in CASSANDRA-12783. 
Here, I only address the issue of holding entire partition into memory when 
repairing base with mv.


was (Author: jasonstack):
[trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13299-trunk]
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-13299 ]

Changes:

1. Throttle by number of base unfiltered. default is 100. 
2. A pair of open/close range tombstone could have any number of unshadowed 
rows in between, in the patch, simply cache the range tombstones to avoid 
exceeding the limit. And apply cached range tombstones, in next batch.

Note:
One partition deletion or a range deletion could cause huge number of view rows 
to be removed, thus view mutation may fail to apply due to WTE or 
max_mutation_size, but it could be resolved separately in CASSANDRA-12783. 
Here, I only address the issue of holding entire partition into memory when 
repairing base with mv.

> Potential OOMs and lock contention in write path streams
> 
>
> Key: CASSANDRA-13299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13299
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Assignee: ZhaoYang
>
> I see a potential OOM, when a stream (e.g. repair) goes through the write 
> path as it is with MVs.
> StreamReceiveTask gets a bunch of SSTableReaders. These produce rowiterators 
> and they again produce mutations. So every partition creates a single 
> mutation, which in case of (very) big partitions can result in (very) big 
> mutations. Those are created on heap and stay there until they finished 
> processing.
> I don't think it is necessary to create a single mutation for each partition. 
> Why don't we implement a PartitionUpdateGeneratorIterator that takes a 
> UnfilteredRowIterator and a max size and spits out PartitionUpdates to be 
> used to create and apply mutations?
> The max size should be something like min(reasonable_absolute_max_size, 
> max_mutation_size, commitlog_segment_size / 2). reasonable_absolute_max_size 
> could be like 16M or sth.
> A mutation shouldn't be too large as it also affects MV partition locking. 
> The longer a MV partition is locked during a stream, the higher chances are 
> that WTE's occur during streams.
> I could also imagine that a max number of updates per mutation regardless of 
> size in bytes could make sense to avoid lock contention.
> Love to get feedback and suggestions, incl. naming suggestions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130152#comment-16130152
 ] 

Romain GERARD edited comment on CASSANDRA-13418 at 8/21/17 11:16 AM:
-

Hi,

I am back with a new proposition 
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05

Majors differences : 
 * Used [~krummas] way for introducing the ignore Overlaps
 * I splitted the function that is doing the overlapingChecks as in the 
previous patch, I was wrongfully checking for overlaps in memtables (even if 
the option was activated) 
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05#diff-e8e282423dcbf34d30a3578c8dec15cdR170
 * I enable uncheckedTombstoneCompaction when ignoreOverlaps is activated  
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05#diff-e83635b2fb3079d9b91b039c605c15daR71
It seems a sane default for me, as even if we drop fully expired sstables, 
we will still check for worth Dropping ones and we want to also ignore overlaps 
check in this case. (was the default behavior in the last patch)
* Added a simple test case. I will look to add more (feel free to suggest some)
* Rebased upon trunk

Every tests passes (ant test) and I will deploy this patch internally to 
confirm that it works as expected.
If you have any remarks [~krummas] in the mean time


was (Author: rgerard):
Hi,

I am back with a new proposition 
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05

Majors differences : 
 + Used [~krummas] way for introducing the ignore Overlaps
 + I splitted the function that is doing the overlapingChecks as in the 
previous patch, I was wrongfully checking for overlaps in memtables (even if 
the option was activated) 
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05#diff-e8e282423dcbf34d30a3578c8dec15cdR170
 + I enable uncheckedTombstoneCompaction when ignoreOverlaps is activated  
https://github.com/criteo-forks/cassandra/commit/0c4d342341340115d2c8d15f78b2cb3eab3c2f05#diff-e83635b2fb3079d9b91b039c605c15daR71
It seems a sane default for me, as even if we drop fully expired sstables, 
we will still check for worth Dropping ones and we want to also ignore overlaps 
check in this case. (was the default behavior in the last patch)
+ Added a simple test case. I will look to add more (feel free to suggest some)
+ Rebased upon trunk

Every tests passes (ant test) and I will deploy this patch internally to 
confirm that it works as expected.
If you have any remarks [~krummas] in the mean time

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: 

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135028#comment-16135028
 ] 

mck edited comment on CASSANDRA-13418 at 8/21/17 10:55 AM:
---

[~rgerard],

Initial review comments: 
 - {{CHANGES.txt}} needs a line
 - is this suitable for 3.11.x as well? (Does it constitute a stability patch? 
i think so, but that's an opinion)
 - is the method {{TWCSCompactionController.getFullyExpiredSSTables(..)}} 
really needed? (CompactionController:170 seems to do what we need already, or 
am i missing something?)
 - {{CompactionController.ignoreOverlaps()}} is a bit of a concept, it deserves 
some apidoc love.

Trivial stuff…
 - CompactionController:108 missing space in {{if(}}, same on :170
 - CompactionController:232 any reason not to return an immutable set?
 - TWCSCompactionController:29 needs a blank line after
 - TWCSCompactionController:33 java declaration order. (static fields go before 
member fields)
 - TimeWindowCompactionStrategy:70  missing space in {{if(}}


was (Author: michaelsembwever):
[~rgerard],

Initial review comments: 
 - {{CHANGES.txt}} needs a line
 - is this suitable for 3.11.x as well? (Does it constitute a stability patch? 
i think so, but that's an opinion)
 - is the method {{TWCSCompactionController.getFullyExpiredSSTables(..)}} 
really needed? (CompactionController:170 seems to do what we need already)
 - {{CompactionController.ignoreOverlaps()}} is a bit of a concept, it deserves 
some apidoc love.

Trivial stuff…
 - CompactionController:108 missing space in {{if(}}, same on :170
 - CompactionController:232 any reason not to return an immutable set?
 - TWCSCompactionController:29 needs a blank line after
 - TWCSCompactionController:33 java declaration order. (static fields go before 
member fields)
 - TimeWindowCompactionStrategy:70  missing space in {{if(}}

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135028#comment-16135028
 ] 

mck commented on CASSANDRA-13418:
-

[~rgerard],

Initial review comments: 
 - {{CHANGES.txt}} needs a line
 - is this suitable for 3.11.x as well? (Does it constitute a stability patch? 
i think so, but that's an opinion)
 - is the method {{TWCSCompactionController.getFullyExpiredSSTables(..)}} 
really needed? (CompactionController:170 seems to do what we need already)
 - {{CompactionController.ignoreOverlaps()}} is a bit of a concept, it deserves 
some apidoc love.

Trivial stuff…
 - CompactionController:108 missing space in {{if(}}, same on :170
 - CompactionController:232 any reason not to return an immutable set?
 - TWCSCompactionController:29 needs a blank line after
 - TWCSCompactionController:33 java declaration order. (static fields go before 
member fields)
 - TimeWindowCompactionStrategy:70  missing space in {{if(}}

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13263) Incorrect ComplexColumnData hashCode implementation

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134941#comment-16134941
 ] 

Sylvain Lebresne commented on CASSANDRA-13263:
--

Sorry that this kind of felt through the cracks. I'm fine changing that thought 
the "correct" fix here is to use {{BTree.hashCode}} here, not 
{{Arrays.hashCode}}. If you change that, I'm +1 on the change, though I'd 
personally stick to just trunk for this as it's definitively good 
future-proofing but, I'm relatively confident, doesn't matter with the current 
code: I don't think that method is called in practice and I see that unlikely 
to change, at least on 3.X. Definitively won't fight it if it makes you sleep 
better to have it in 3.0/3.11 though for basically the exact same reasons (and 
it's a pretty safe change anyway).

> Incorrect ComplexColumnData hashCode implementation
> ---
>
> Key: CASSANDRA-13263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13263
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>
> I went through some of the logs from CASSANDRA-13175 and one of the more 
> serious issues that we should address seem to be the 
> {{ComplexColumnData.hashCode()}} implementation. As Objects.hashCode is not 
> using deep hashing for array arguments, hashed will be based on the identity 
> instead of the array's content. See patch for simple fix.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline

[Norman Maurer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134918#comment-16134918
 ] 

Norman Maurer commented on CASSANDRA-13649:
---

Its up to you guys which versions the patch should be applied to, just wanted 
to mention this fix is very low-risk in terms of Netty itself as it just move 
logic to an extra ChannelHandler. Thats all. 

> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
>  Labels: patch
> Attachments: 
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch, 
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
>  Just want to make sure that we don't have to change anything related to the 
> exception handling in our pipeline and that this isn't a netty issue. 
> Actually if this causes flakiness but is otherwise harmless, we should do 
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() 
> failed}} error also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13215) Cassandra nodes startup time 20x more after upgarding to 3.x

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13215:

Reviewer:   (was: Marcus Eriksson)

> Cassandra nodes startup time 20x more after upgarding to 3.x
> 
>
> Key: CASSANDRA-13215
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13215
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Cluster setup: two datacenters (dc-main, dc-backup).
> dc-main - 9 servers, no vnodes
> dc-backup - 6 servers, vnodes
>Reporter: Viktor Kuzmin
>Assignee: Marcus Eriksson
> Attachments: simple-cache.patch
>
>
> CompactionStrategyManage.getCompactionStrategyIndex is called on each sstable 
> at startup. And this function calls StorageService.getDiskBoundaries. And 
> getDiskBoundaries calls AbstractReplicationStrategy.getAddressRanges.
> It appears that last function can be really slow. In our environment we have 
> 1545 tokens and with NetworkTopologyStrategy it can make 1545*1545 
> computations in worst case (maybe I'm wrong, but it really takes lot's of 
> cpu).
> Also this function can affect runtime later, cause it is called not only 
> during startup.
> I've tried to implement simple cache for getDiskBoundaries results and now 
> startup time is about one minute instead of 25m, but I'm not sure if it's a 
> good solution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12938) cassandra-stress hangs on error

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-12938:
-
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   4.0
   3.11.1
   Status: Resolved  (was: Patch Available)

Committed to 3.11 as {{1619413e5602b93641af1ccf5adbee80eaa2b53c}} and merged 
into trunk.

> cassandra-stress hangs on error
> ---
>
> Key: CASSANDRA-12938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12938
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: James Falcon
>Assignee: Eduard Tudenhoefner
> Fix For: 3.11.1, 4.0
>
>
> After encountering a fatal error, cassandra-stress hangs. Having not run a 
> previous stress write, can be reproduced with:
> {code}
> cassandra-stress read n=1000 -rate threads=2
> {code}
> Here's the full output
> {code}
>  Stress Settings 
> Command:
>   Type: read
>   Count: 1,000
>   No Warmup: false
>   Consistency Level: LOCAL_ONE
>   Target Uncertainty: not applicable
>   Key Size (bytes): 10
>   Counter Increment Distibution: add=fixed(1)
> Rate:
>   Auto: false
>   Thread Count: 2
>   OpsPer Sec: 0
> Population:
>   Distribution: Gaussian:  min=1,max=1000,mean=500.50,stdev=166.50
>   Order: ARBITRARY
>   Wrap: false
> Insert:
>   Revisits: Uniform:  min=1,max=100
>   Visits: Fixed:  key=1
>   Row Population Ratio: Ratio: divisor=1.00;delegate=Fixed:  key=1
>   Batch Type: not batching
> Columns:
>   Max Columns Per Key: 5
>   Column Names: [C0, C1, C2, C3, C4]
>   Comparator: AsciiType
>   Timestamp: null
>   Variable Column Count: false
>   Slice: false
>   Size Distribution: Fixed:  key=34
>   Count Distribution: Fixed:  key=5
> Errors:
>   Ignore: false
>   Tries: 10
> Log:
>   No Summary: false
>   No Settings: false
>   File: null
>   Interval Millis: 1000
>   Level: NORMAL
> Mode:
>   API: JAVA_DRIVER_NATIVE
>   Connection Style: CQL_PREPARED
>   CQL Version: CQL3
>   Protocol Version: V4
>   Username: null
>   Password: null
>   Auth Provide Class: null
>   Max Pending Per Connection: 128
>   Connections Per Host: 8
>   Compression: NONE
> Node:
>   Nodes: [localhost]
>   Is White List: false
>   Datacenter: null
> Schema:
>   Keyspace: keyspace1
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>   Replication Strategy Pptions: {replication_factor=1}
>   Table Compression: null
>   Table Compaction Strategy: null
>   Table Compaction Strategy Options: {}
> Transport:
>   factory=org.apache.cassandra.thrift.TFramedTransportFactory; 
> truststore=null; truststore-password=null; keystore=null; 
> keystore-password=null; ssl-protocol=TLS; ssl-alg=SunX509; store-type=JKS; 
> ssl-ciphers=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA; 
> Port:
>   Native Port: 9042
>   Thrift Port: 9160
>   JMX Port: 9042
> Send To Daemon:
>   *not set*
> Graph:
>   File: null
>   Revision: unknown
>   Title: null
>   Operation: READ
> TokenRange:
>   Wrap: false
>   Split Factor: 1
> Sleeping 2s...
> Warming up READ with 250 iterations...
> Connected to cluster: falcon-test2, max pending requests per connection 128, 
> max connections per host 8
> Datatacenter: Cassandra; Host: localhost/127.0.0.1; Rack: rack1
> Failed to connect over JMX; not collecting these stats
> Connected to cluster: falcon-test2, max pending requests per connection 128, 
> max connections per host 8
> Datatacenter: Cassandra; Host: localhost/127.0.0.1; Rack: rack1
> com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 
> 'keyspace1' does not exist
> Connected to cluster: falcon-test2, max pending requests per connection 128, 
> max connections per host 8
> Datatacenter: Cassandra; Host: localhost/127.0.0.1; Rack: rack1
> com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 
> 'keyspace1' does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[stefania]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6e42dd21
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6e42dd21
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6e42dd21

Branch: refs/heads/trunk
Commit: 6e42dd215cd55b933691379ee5f96fbdeea9cc78
Parents: b740efa 1619413
Author: Stefania Alborghetti 
Authored: Mon Aug 21 16:35:50 2017 +0800
Committer: Stefania Alborghetti 
Committed: Mon Aug 21 16:35:50 2017 +0800

--
 CHANGES.txt |  1 +
 .../apache/cassandra/stress/StressAction.java   | 32 +++-
 2 files changed, 19 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6e42dd21/CHANGES.txt
--
diff --cc CHANGES.txt
index 1f67f3b,48c21cc..9ba081f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,124 -1,5 +1,125 @@@
 +4.0
 + * Add bytes repaired/unrepaired to nodetool tablestats (CASSANDRA-13774)
 + * Don't delete incremental repair sessions if they still have sstables 
(CASSANDRA-13758)
 + * Fix pending repair manager index out of bounds check (CASSANDRA-13769)
 + * Don't use RangeFetchMapCalculator when RF=1 (CASSANDRA-13576)
 + * Don't optimise trivial ranges in RangeFetchMapCalculator (CASSANDRA-13664)
 + * Use an ExecutorService for repair commands instead of new 
Thread(..).start() (CASSANDRA-13594)
 + * Fix race / ref leak in anticompaction (CASSANDRA-13688)
 + * Expose tasks queue length via JMX (CASSANDRA-12758)
 + * Fix race / ref leak in PendingRepairManager (CASSANDRA-13751)
 + * Enable ppc64le runtime as unsupported architecture (CASSANDRA-13615)
 + * Improve sstablemetadata output (CASSANDRA-11483)
 + * Support for migrating legacy users to roles has been dropped 
(CASSANDRA-13371)
 + * Introduce error metrics for repair (CASSANDRA-13387)
 + * Refactoring to primitive functional interfaces in AuthCache 
(CASSANDRA-13732)
 + * Update metrics to 3.1.5 (CASSANDRA-13648)
 + * batch_size_warn_threshold_in_kb can now be set at runtime (CASSANDRA-13699)
 + * Avoid always rebuilding secondary indexes at startup (CASSANDRA-13725)
 + * Upgrade JMH from 1.13 to 1.19 (CASSANDRA-13727)
 + * Upgrade SLF4J from 1.7.7 to 1.7.25 (CASSANDRA-12996)
 + * Default for start_native_transport now true if not set in config 
(CASSANDRA-13656)
 + * Don't add localhost to the graph when calculating where to stream from 
(CASSANDRA-13583)
 + * Allow skipping equality-restricted clustering columns in ORDER BY clause 
(CASSANDRA-10271)
 + * Use common nowInSec for validation compactions (CASSANDRA-13671)
 + * Improve handling of IR prepare failures (CASSANDRA-13672)
 + * Send IR coordinator messages synchronously (CASSANDRA-13673)
 + * Flush system.repair table before IR finalize promise (CASSANDRA-13660)
 + * Fix column filter creation for wildcard queries (CASSANDRA-13650)
 + * Add 'nodetool getbatchlogreplaythrottle' and 'nodetool 
setbatchlogreplaythrottle' (CASSANDRA-13614)
 + * fix race condition in PendingRepairManager (CASSANDRA-13659)
 + * Allow noop incremental repair state transitions (CASSANDRA-13658)
 + * Run repair with down replicas (CASSANDRA-10446)
 + * Added started & completed repair metrics (CASSANDRA-13598)
 + * Added started & completed repair metrics (CASSANDRA-13598)
 + * Improve secondary index (re)build failure and concurrency handling 
(CASSANDRA-10130)
 + * Improve calculation of available disk space for compaction 
(CASSANDRA-13068)
 + * Change the accessibility of RowCacheSerializer for third party row cache 
plugins (CASSANDRA-13579)
 + * Allow sub-range repairs for a preview of repaired data (CASSANDRA-13570)
 + * NPE in IR cleanup when columnfamily has no sstables (CASSANDRA-13585)
 + * Fix Randomness of stress values (CASSANDRA-12744)
 + * Allow selecting Map values and Set elements (CASSANDRA-7396)
 + * Fast and garbage-free Streaming Histogram (CASSANDRA-13444)
 + * Update repairTime for keyspaces on completion (CASSANDRA-13539)
 + * Add configurable upper bound for validation executor threads 
(CASSANDRA-13521)
 + * Bring back maxHintTTL propery (CASSANDRA-12982)
 + * Add testing guidelines (CASSANDRA-13497)
 + * Add more repair metrics (CASSANDRA-13531)
 + * RangeStreamer should be smarter when picking endpoints for streaming 
(CASSANDRA-4650)
 + * Avoid rewrapping an exception thrown for cache load functions 
(CASSANDRA-13367)
 + * Log time elapsed for each incremental repair phase (CASSANDRA-13498)
 + * Add multiple table operation support to cassandra-stress (CASSANDRA-8780)
 + * Fix incorrect cqlsh results when selecting same columns multiple times 
(CASSANDRA-13262)
 + * Fix 

[2/3] cassandra git commit: Fix cassandra-stress hang issues when an error during cluster connection happens

[stefania]

Fix cassandra-stress hang issues when an error during cluster connection happens

patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for 
CASSANDRA-12938


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1619413e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1619413e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1619413e

Branch: refs/heads/trunk
Commit: 1619413e5602b93641af1ccf5adbee80eaa2b53c
Parents: 01f2855
Author: Eduard Tudenhoefner 
Authored: Mon Aug 21 11:36:08 2017 +0800
Committer: Stefania Alborghetti 
Committed: Mon Aug 21 11:36:25 2017 +0800

--
 CHANGES.txt |  1 +
 .../apache/cassandra/stress/StressAction.java   | 40 +++-
 2 files changed, 23 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1619413e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4ede932..48c21cc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.1
+ * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
  * Better bootstrap failure message when blocked by (potential) range movement 
(CASSANDRA-13744)
  * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
  * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1619413e/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java 
b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index 30f8899..5a340e8 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -420,27 +420,31 @@ public class StressAction implements Runnable
 SimpleClient sclient = null;
 ThriftClient tclient = null;
 JavaDriverClient jclient = null;
-
-
 final ConnectionAPI clientType = settings.mode.api;
-switch (clientType)
+
+try
 {
-case JAVA_DRIVER_NATIVE:
-jclient = settings.getJavaDriverClient();
-break;
-case SIMPLE_NATIVE:
-sclient = settings.getSimpleNativeClient();
-break;
-case THRIFT:
-case THRIFT_SMART:
-tclient = settings.getThriftClient();
-break;
-default:
-throw new IllegalStateException();
+switch (clientType)
+{
+case JAVA_DRIVER_NATIVE:
+jclient = settings.getJavaDriverClient();
+break;
+case SIMPLE_NATIVE:
+sclient = settings.getSimpleNativeClient();
+break;
+case THRIFT:
+case THRIFT_SMART:
+tclient = settings.getThriftClient();
+break;
+default:
+throw new IllegalStateException();
+}
+}
+finally
+{
+// synchronize the start of all the consumer threads
+start.countDown();
 }
-
-// synchronize the start of all the consumer threads
-start.countDown();
 
 releaseConsumers.await();
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/3] cassandra git commit: Fix cassandra-stress hang issues when an error during cluster connection happens

[stefania]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 01f2855b4 -> 1619413e5
  refs/heads/trunk b740efa73 -> 6e42dd215


Fix cassandra-stress hang issues when an error during cluster connection happens

patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for 
CASSANDRA-12938


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1619413e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1619413e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1619413e

Branch: refs/heads/cassandra-3.11
Commit: 1619413e5602b93641af1ccf5adbee80eaa2b53c
Parents: 01f2855
Author: Eduard Tudenhoefner 
Authored: Mon Aug 21 11:36:08 2017 +0800
Committer: Stefania Alborghetti 
Committed: Mon Aug 21 11:36:25 2017 +0800

--
 CHANGES.txt |  1 +
 .../apache/cassandra/stress/StressAction.java   | 40 +++-
 2 files changed, 23 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1619413e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4ede932..48c21cc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.1
+ * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
  * Better bootstrap failure message when blocked by (potential) range movement 
(CASSANDRA-13744)
  * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
  * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1619413e/tools/stress/src/org/apache/cassandra/stress/StressAction.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java 
b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index 30f8899..5a340e8 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -420,27 +420,31 @@ public class StressAction implements Runnable
 SimpleClient sclient = null;
 ThriftClient tclient = null;
 JavaDriverClient jclient = null;
-
-
 final ConnectionAPI clientType = settings.mode.api;
-switch (clientType)
+
+try
 {
-case JAVA_DRIVER_NATIVE:
-jclient = settings.getJavaDriverClient();
-break;
-case SIMPLE_NATIVE:
-sclient = settings.getSimpleNativeClient();
-break;
-case THRIFT:
-case THRIFT_SMART:
-tclient = settings.getThriftClient();
-break;
-default:
-throw new IllegalStateException();
+switch (clientType)
+{
+case JAVA_DRIVER_NATIVE:
+jclient = settings.getJavaDriverClient();
+break;
+case SIMPLE_NATIVE:
+sclient = settings.getSimpleNativeClient();
+break;
+case THRIFT:
+case THRIFT_SMART:
+tclient = settings.getThriftClient();
+break;
+default:
+throw new IllegalStateException();
+}
+}
+finally
+{
+// synchronize the start of all the consumer threads
+start.countDown();
 }
-
-// synchronize the start of all the consumer threads
-start.countDown();
 
 releaseConsumers.await();
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-12711) dtest failure in compaction_test.TestCompaction_with_DateTieredCompactionStrategy.bloomfilter_size_test

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson resolved CASSANDRA-12711.
-
Resolution: Cannot Reproduce

looks like this is now stable:
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/lastCompletedBuild/testReport/compaction_test/TestCompaction_with_DateTieredCompactionStrategy/bloomfilter_size_test/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-3.0-dtest/147/testReport/compaction_test/TestCompaction_with_DateTieredCompactionStrategy/bloomfilter_size_test/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-3.11-dtest/170/testReport/compaction_test/TestCompaction_with_DateTieredCompactionStrategy/bloomfilter_size_test/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/lastCompletedBuild/testReport/compaction_test/TestCompaction_with_DateTieredCompactionStrategy/bloomfilter_size_test/

> dtest failure in 
> compaction_test.TestCompaction_with_DateTieredCompactionStrategy.bloomfilter_size_test
> ---
>
> Key: CASSANDRA-12711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12711
> Project: Cassandra
>  Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Marcus Eriksson
>  Labels: dtest
> Attachments: node1_debug.log, node1_gc.log, node1.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/485/testReport/compaction_test/TestCompaction_with_DateTieredCompactionStrategy/bloomfilter_size_test
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/compaction_test.py", line 153, in 
> bloomfilter_size_test
> self.assertLessEqual(bfSize, size_factor * max_bf_size)
>   File "/usr/lib/python2.7/unittest/case.py", line 936, in assertLessEqual
> self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "457864 not less than or equal to 40.0
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134827#comment-16134827
 ] 

Stefan Podkowinski commented on CASSANDRA-13649:


+1 for 2.2 upwards. If the fix seems to be too risky for 2.2, we should 
probably not include it in 3.0 as well. On the opposite, if it's not, then 
let's not diverge our code base without good reason.

> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
>  Labels: patch
> Attachments: 
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch, 
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
>  Just want to make sure that we don't have to change anything related to the 
> exception handling in our pipeline and that this isn't a netty issue. 
> Actually if this causes flakiness but is otherwise harmless, we should do 
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 
> - An exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handle 
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
> Connection reset by peer
>   at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() 
> failed}} error also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134811#comment-16134811
 ] 

Kurt Greaves commented on CASSANDRA-12783:
--

Kind of. Was actually working on fixing CASSANDRA-13608 which is somewhat 
related, following Paulo's ideas from 
[here|https://issues.apache.org/jira/browse/CASSANDRA-11670?focusedCommentId=15359128=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15359128]
 to fix the limitation on the batchlog of only being able to contain 
{{max_value_size}}. This also introduces the concept of "activating" a batch 
mentioned in the description. Not really sure that it will prevent OOMs, but 
description isn't really clear what that is about. I will upload my branch 
somewhere shortly, have actually gotten sidetracked trying to work out why I 
can generate time based UUIDs that are not ordered on time... but more on that 
later



> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13299) Potential OOMs and lock contention in write path streams

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134749#comment-16134749
 ] 

Benjamin Roth commented on CASSANDRA-13299:
---

Sorry for the late response, I was on vacation. No, I am not working on that 
ticket. But thanks a lot for your efforts (not only) on that ticket!

> Potential OOMs and lock contention in write path streams
> 
>
> Key: CASSANDRA-13299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13299
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Assignee: ZhaoYang
>
> I see a potential OOM, when a stream (e.g. repair) goes through the write 
> path as it is with MVs.
> StreamReceiveTask gets a bunch of SSTableReaders. These produce rowiterators 
> and they again produce mutations. So every partition creates a single 
> mutation, which in case of (very) big partitions can result in (very) big 
> mutations. Those are created on heap and stay there until they finished 
> processing.
> I don't think it is necessary to create a single mutation for each partition. 
> Why don't we implement a PartitionUpdateGeneratorIterator that takes a 
> UnfilteredRowIterator and a max size and spits out PartitionUpdates to be 
> used to create and apply mutations?
> The max size should be something like min(reasonable_absolute_max_size, 
> max_mutation_size, commitlog_segment_size / 2). reasonable_absolute_max_size 
> could be like 16M or sth.
> A mutation shouldn't be too large as it also affects MV partition locking. 
> The longer a MV partition is locked during a stream, the higher chances are 
> that WTE's occur during streams.
> I could also imagine that a max number of updates per mutation regardless of 
> size in bytes could make sense to avoid lock contention.
> Love to get feedback and suggestions, incl. naming suggestions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org