[jira] [Commented] (CASSANDRA-14027) consistent_range_movement_false_with_rf1_should_succeed_test - bootstrap_test.TestBootstrap

[Vincent White (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279587#comment-16279587
 ] 

Vincent White commented on CASSANDRA-14027:
---

I had this test running against trunk for a few hours and didn't see any 
failures.

> consistent_range_movement_false_with_rf1_should_succeed_test - 
> bootstrap_test.TestBootstrap
> ---
>
> Key: CASSANDRA-14027
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14027
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>
> 15 Nov 2017 11:22:19 [node2] Timed out waiting for 
> /tmp/dtest-3SeOAb/test/node2/logs/system.log to be created.
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-3SeOAb
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/cassandra/cassandra-dtest/bootstrap_test.py", line 239, in 
> consistent_range_movement_false_with_rf1_should_succeed_test
> self._bootstrap_test_with_replica_down(False, rf=1)
>   File "/home/cassandra/cassandra-dtest/bootstrap_test.py", line 261, in 
> _bootstrap_test_with_replica_down
> cluster.start()
>   File 
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py", 
> line 415, in start
> node.watch_log_for(start_message, timeout=kwargs.get('timeout',60), 
> process=p, verbose=verbose, from_mark=mark)
>   File 
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/node.py", line 
> 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " 
> [" + self.name + "] Timed out waiting for {} to be created.".format(log_file))
> "15 Nov 2017 11:22:19 [node2] Timed out waiting for 
> /tmp/dtest-3SeOAb/test/node2/logs/system.log to be 
> created.\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /tmp/dtest-3SeOAb\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\n- >> end captured logging << 
> -"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14071) Materialized view on table with TTL issue

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279554#comment-16279554
 ] 

ZhaoYang commented on CASSANDRA-14071:
--

LGTM, thanks for the review

> Materialized view on table with TTL issue
> -
>
> Key: CASSANDRA-14071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14071
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination, Materialized Views
> Environment: Cassandra 3
>Reporter: Silviu Butnariu
>Assignee: ZhaoYang
>  Labels: correctness
>
> Materialized views that cluster by a column that is not part of table's PK 
> and are created from tables that have *default_time_to_live* seems to 
> malfunction.
> Having this table
> {code:java}
> CREATE TABLE sbutnariu.test_bug (
> field1 smallint,
> field2 smallint,
> date timestamp,
> PRIMARY KEY ((field1), field2)
> ) WITH default_time_to_live = 1000;
> {code}
> and the materialized view
> {code:java}
> CREATE MATERIALIZED VIEW sbutnariu.test_bug_by_date AS SELECT * FROM 
> sbutnariu.test_bug WHERE field1 IS NOT NULL AND field2 IS NOT NULL AND date 
> IS NOT NULL PRIMARY KEY ((field1), date, field2) WITH CLUSTERING ORDER BY 
> (date desc, field2 asc);
> {code}
> After inserting 3 rows with same PK (should upsert), the materialized view 
> will have 3 rows.
> {code:java}
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*3 rows*/
> {code}
> If I remove the ttl and try again, it works as expected:
> {code:java}
> truncate sbutnariu.test_bug;
> alter table sbutnariu.test_bug with default_time_to_live = 0;
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*1 row*/
> {code}
> I've tested on versions 3.0.14 and 3.0.15. The bug was introduced in 3.0.15, 
> as in 3.0.14 it works as expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14071) Materialized view on table with TTL issue

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279552#comment-16279552
 ] 

Paulo Motta commented on CASSANDRA-14071:
-

Patch and tests look good, I just made this following minor changes:
* Improve docs a bit to give more information on how {{ExpiredLivenessInfo}} is 
used 
[here|https://github.com/pauloricardomg/cassandra/commit/0098a04f50db06266ecd54cd115de8b167c38862]
* Avoid unnecessary comparison on 
[LivenessInfo.supersedes|https://github.com/pauloricardomg/cassandra/commit/d4eafba373973c870158bcb2b005746f9f59aa2d].
* Reinstate [removed 
assertion|https://github.com/pauloricardomg/cassandra/commit/7a79e0837004415885b8cddaf2f45ba21d493bc3]

I rebased submitted CI with these changes:
* [3.0|https://github.com/pauloricardomg/cassandra/tree/CASSANDRA-14071]
* [3.11|https://github.com/pauloricardomg/cassandra/tree/CASSANDRA-14071-3.11]
* [trunk|https://github.com/pauloricardomg/cassandra/tree/CASSANDRA-14071-trunk]

Please let me know what do you think. 

> Materialized view on table with TTL issue
> -
>
> Key: CASSANDRA-14071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14071
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination, Materialized Views
> Environment: Cassandra 3
>Reporter: Silviu Butnariu
>Assignee: ZhaoYang
>  Labels: correctness
>
> Materialized views that cluster by a column that is not part of table's PK 
> and are created from tables that have *default_time_to_live* seems to 
> malfunction.
> Having this table
> {code:java}
> CREATE TABLE sbutnariu.test_bug (
> field1 smallint,
> field2 smallint,
> date timestamp,
> PRIMARY KEY ((field1), field2)
> ) WITH default_time_to_live = 1000;
> {code}
> and the materialized view
> {code:java}
> CREATE MATERIALIZED VIEW sbutnariu.test_bug_by_date AS SELECT * FROM 
> sbutnariu.test_bug WHERE field1 IS NOT NULL AND field2 IS NOT NULL AND date 
> IS NOT NULL PRIMARY KEY ((field1), date, field2) WITH CLUSTERING ORDER BY 
> (date desc, field2 asc);
> {code}
> After inserting 3 rows with same PK (should upsert), the materialized view 
> will have 3 rows.
> {code:java}
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*3 rows*/
> {code}
> If I remove the ttl and try again, it works as expected:
> {code:java}
> truncate sbutnariu.test_bug;
> alter table sbutnariu.test_bug with default_time_to_live = 0;
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*1 row*/
> {code}
> I've tested on versions 3.0.14 and 3.0.15. The bug was introduced in 3.0.15, 
> as in 3.0.14 it works as expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14071) Materialized view on table with TTL issue

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279513#comment-16279513
 ] 

ZhaoYang commented on CASSANDRA-14071:
--

CI looks good. There is some NPE on view unit test while creating keyspace, but 
it looks related to CASSANDRA-14010

> Materialized view on table with TTL issue
> -
>
> Key: CASSANDRA-14071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14071
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination, Materialized Views
> Environment: Cassandra 3
>Reporter: Silviu Butnariu
>Assignee: ZhaoYang
>  Labels: correctness
>
> Materialized views that cluster by a column that is not part of table's PK 
> and are created from tables that have *default_time_to_live* seems to 
> malfunction.
> Having this table
> {code:java}
> CREATE TABLE sbutnariu.test_bug (
> field1 smallint,
> field2 smallint,
> date timestamp,
> PRIMARY KEY ((field1), field2)
> ) WITH default_time_to_live = 1000;
> {code}
> and the materialized view
> {code:java}
> CREATE MATERIALIZED VIEW sbutnariu.test_bug_by_date AS SELECT * FROM 
> sbutnariu.test_bug WHERE field1 IS NOT NULL AND field2 IS NOT NULL AND date 
> IS NOT NULL PRIMARY KEY ((field1), date, field2) WITH CLUSTERING ORDER BY 
> (date desc, field2 asc);
> {code}
> After inserting 3 rows with same PK (should upsert), the materialized view 
> will have 3 rows.
> {code:java}
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*3 rows*/
> {code}
> If I remove the ttl and try again, it works as expected:
> {code:java}
> truncate sbutnariu.test_bug;
> alter table sbutnariu.test_bug with default_time_to_live = 0;
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*1 row*/
> {code}
> I've tested on versions 3.0.14 and 3.0.15. The bug was introduced in 3.0.15, 
> as in 3.0.14 it works as expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14071) Materialized view on table with TTL issue

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279510#comment-16279510
 ] 

ZhaoYang commented on CASSANDRA-14071:
--

It will be in latest bugfix version: 3.0.x/3.11.x/4.0

> Materialized view on table with TTL issue
> -
>
> Key: CASSANDRA-14071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14071
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination, Materialized Views
> Environment: Cassandra 3
>Reporter: Silviu Butnariu
>Assignee: ZhaoYang
>  Labels: correctness
>
> Materialized views that cluster by a column that is not part of table's PK 
> and are created from tables that have *default_time_to_live* seems to 
> malfunction.
> Having this table
> {code:java}
> CREATE TABLE sbutnariu.test_bug (
> field1 smallint,
> field2 smallint,
> date timestamp,
> PRIMARY KEY ((field1), field2)
> ) WITH default_time_to_live = 1000;
> {code}
> and the materialized view
> {code:java}
> CREATE MATERIALIZED VIEW sbutnariu.test_bug_by_date AS SELECT * FROM 
> sbutnariu.test_bug WHERE field1 IS NOT NULL AND field2 IS NOT NULL AND date 
> IS NOT NULL PRIMARY KEY ((field1), date, field2) WITH CLUSTERING ORDER BY 
> (date desc, field2 asc);
> {code}
> After inserting 3 rows with same PK (should upsert), the materialized view 
> will have 3 rows.
> {code:java}
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*3 rows*/
> {code}
> If I remove the ttl and try again, it works as expected:
> {code:java}
> truncate sbutnariu.test_bug;
> alter table sbutnariu.test_bug with default_time_to_live = 0;
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*1 row*/
> {code}
> I've tested on versions 3.0.14 and 3.0.15. The bug was introduced in 3.0.15, 
> as in 3.0.14 it works as expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14091) DynamicSnitch creates a lot of garbage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279458#comment-16279458
 ] 

ASF GitHub Bot commented on CASSANDRA-14091:


Github user jeffjirsa commented on the issue:

https://github.com/apache/cassandra/pull/178
  
Hi @Benmartin92 , That looks like a perfectly reasonable patch. It appears 
that @bdeggleston also noticed this in 
https://issues.apache.org/jira/browse/CASSANDRA-14091 (a week or so after your 
patch). Unfortunately, I don't think anyone noticed your patch, probably 
because we don't typically use Github PRs (ASF repo isn't writable on github, 
so we can't merge this PR directly) - our contribution workflow is through 
JIRA. Appreciate the contribution, sorry we didn't see it in time. 


> DynamicSnitch creates a lot of garbage
> --
>
> Key: CASSANDRA-14091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14091
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The ExponentiallyDecayingReservoir snapshots we take during score updates 
> generate a lot of garbage, and we call getSnapshot twice per endpoint when we 
> only need to call it once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14091) DynamicSnitch creates a lot of garbage

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-14091:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Thanks, committed as {{10ca7e47ca63c43b4e0ba593fb4c736130764af9}}

> DynamicSnitch creates a lot of garbage
> --
>
> Key: CASSANDRA-14091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14091
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The ExponentiallyDecayingReservoir snapshots we take during score updates 
> generate a lot of garbage, and we call getSnapshot twice per endpoint when we 
> only need to call it once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[bdeggleston]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ae782319
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ae782319
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ae782319

Branch: refs/heads/cassandra-3.11
Commit: ae782319b59266080cc85b3fd3210cdf39a4353a
Parents: c3a1a4f 10ca7e4
Author: Blake Eggleston 
Authored: Tue Dec 5 16:06:15 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:06:15 2017 -0800

--
 CHANGES.txt |  2 ++
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 14 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae782319/CHANGES.txt
--
diff --cc CHANGES.txt
index 7c0af91,cf8883a..60215c4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,12 -1,5 +1,14 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+ 3.0.16
+  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae782319/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index e9c8e6f,8c255f5..42fc26c
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@@ -30,12 -29,9 +30,13 @@@ import com.codahale.metrics.Exponential
  import javax.management.MBeanServer;
  import javax.management.ObjectName;
  
+ import com.codahale.metrics.Snapshot;
  import org.apache.cassandra.concurrent.ScheduledExecutors;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.gms.ApplicationState;
 +import org.apache.cassandra.gms.EndpointState;
 +import org.apache.cassandra.gms.Gossiper;
 +import org.apache.cassandra.gms.VersionedValue;
  import org.apache.cassandra.net.MessagingService;
  import org.apache.cassandra.service.StorageService;
  import org.apache.cassandra.utils.FBUtilities;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Reduce garbage created by DynamicSnitch

[bdeggleston]

Reduce garbage created by DynamicSnitch

Patch by Blake Eggleston; Reviewed by Jason Brown for CASSANDRA-14091


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10ca7e47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10ca7e47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10ca7e47

Branch: refs/heads/cassandra-3.11
Commit: 10ca7e47ca63c43b4e0ba593fb4c736130764af9
Parents: 05cb556
Author: Blake Eggleston 
Authored: Fri Dec 1 15:04:12 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:05:07 2017 -0800

--
 CHANGES.txt |  1 +
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 13 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2683dc2..cf8883a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 9c0c57e..8c255f5 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -29,6 +29,7 @@ import com.codahale.metrics.ExponentiallyDecayingReservoir;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 
+import com.codahale.metrics.Snapshot;
 import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.net.MessagingService;
@@ -265,19 +266,26 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 
 }
 double maxLatency = 1;
+
+Map snapshots = new HashMap<>(samples.size());
+for (Map.Entry entry : 
samples.entrySet())
+{
+snapshots.put(entry.getKey(), entry.getValue().getSnapshot());
+}
+
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
 HashMap newScores = new HashMap<>();
-for (Map.Entry entry : 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double mean = entry.getValue().getSnapshot().getMedian();
+double mean = entry.getValue().getMedian();
 if (mean > maxLatency)
 maxLatency = mean;
 }
 // now make another pass to do the weighting based on the maximums we 
found before
-for (Map.Entry entry: 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double score = entry.getValue().getSnapshot().getMedian() / 
maxLatency;
+double score = entry.getValue().getMedian() / maxLatency;
 // finally, add the severity without any weighting, since hosts 
scale this relative to their own load and the size of the task causing the 
severity.
 // "Severity" is basically a measure of compaction activity 
(CASSANDRA-3722).
 if (USE_SEVERITY)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Reduce garbage created by DynamicSnitch

[bdeggleston]

Reduce garbage created by DynamicSnitch

Patch by Blake Eggleston; Reviewed by Jason Brown for CASSANDRA-14091


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10ca7e47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10ca7e47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10ca7e47

Branch: refs/heads/trunk
Commit: 10ca7e47ca63c43b4e0ba593fb4c736130764af9
Parents: 05cb556
Author: Blake Eggleston 
Authored: Fri Dec 1 15:04:12 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:05:07 2017 -0800

--
 CHANGES.txt |  1 +
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 13 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2683dc2..cf8883a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 9c0c57e..8c255f5 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -29,6 +29,7 @@ import com.codahale.metrics.ExponentiallyDecayingReservoir;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 
+import com.codahale.metrics.Snapshot;
 import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.net.MessagingService;
@@ -265,19 +266,26 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 
 }
 double maxLatency = 1;
+
+Map snapshots = new HashMap<>(samples.size());
+for (Map.Entry entry : 
samples.entrySet())
+{
+snapshots.put(entry.getKey(), entry.getValue().getSnapshot());
+}
+
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
 HashMap newScores = new HashMap<>();
-for (Map.Entry entry : 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double mean = entry.getValue().getSnapshot().getMedian();
+double mean = entry.getValue().getMedian();
 if (mean > maxLatency)
 maxLatency = mean;
 }
 // now make another pass to do the weighting based on the maximums we 
found before
-for (Map.Entry entry: 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double score = entry.getValue().getSnapshot().getMedian() / 
maxLatency;
+double score = entry.getValue().getMedian() / maxLatency;
 // finally, add the severity without any weighting, since hosts 
scale this relative to their own load and the size of the task causing the 
severity.
 // "Severity" is basically a measure of compaction activity 
(CASSANDRA-3722).
 if (USE_SEVERITY)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[bdeggleston]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ae782319
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ae782319
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ae782319

Branch: refs/heads/trunk
Commit: ae782319b59266080cc85b3fd3210cdf39a4353a
Parents: c3a1a4f 10ca7e4
Author: Blake Eggleston 
Authored: Tue Dec 5 16:06:15 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:06:15 2017 -0800

--
 CHANGES.txt |  2 ++
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 14 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae782319/CHANGES.txt
--
diff --cc CHANGES.txt
index 7c0af91,cf8883a..60215c4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,12 -1,5 +1,14 @@@
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+ 3.0.16
+  * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
   * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae782319/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index e9c8e6f,8c255f5..42fc26c
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@@ -30,12 -29,9 +30,13 @@@ import com.codahale.metrics.Exponential
  import javax.management.MBeanServer;
  import javax.management.ObjectName;
  
+ import com.codahale.metrics.Snapshot;
  import org.apache.cassandra.concurrent.ScheduledExecutors;
  import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.gms.ApplicationState;
 +import org.apache.cassandra.gms.EndpointState;
 +import org.apache.cassandra.gms.Gossiper;
 +import org.apache.cassandra.gms.VersionedValue;
  import org.apache.cassandra.net.MessagingService;
  import org.apache.cassandra.service.StorageService;
  import org.apache.cassandra.utils.FBUtilities;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Reduce garbage created by DynamicSnitch

[bdeggleston]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 05cb556f9 -> 10ca7e47c
  refs/heads/cassandra-3.11 c3a1a4fa8 -> ae782319b
  refs/heads/trunk d8f036122 -> ed0ded123


Reduce garbage created by DynamicSnitch

Patch by Blake Eggleston; Reviewed by Jason Brown for CASSANDRA-14091


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10ca7e47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10ca7e47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10ca7e47

Branch: refs/heads/cassandra-3.0
Commit: 10ca7e47ca63c43b4e0ba593fb4c736130764af9
Parents: 05cb556
Author: Blake Eggleston 
Authored: Fri Dec 1 15:04:12 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:05:07 2017 -0800

--
 CHANGES.txt |  1 +
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 13 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2683dc2..cf8883a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Reduce garbage created by DynamicSnitch (CASSANDRA-14091)
  * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ca7e47/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 9c0c57e..8c255f5 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -29,6 +29,7 @@ import com.codahale.metrics.ExponentiallyDecayingReservoir;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 
+import com.codahale.metrics.Snapshot;
 import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.net.MessagingService;
@@ -265,19 +266,26 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 
 }
 double maxLatency = 1;
+
+Map snapshots = new HashMap<>(samples.size());
+for (Map.Entry entry : 
samples.entrySet())
+{
+snapshots.put(entry.getKey(), entry.getValue().getSnapshot());
+}
+
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
 HashMap newScores = new HashMap<>();
-for (Map.Entry entry : 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double mean = entry.getValue().getSnapshot().getMedian();
+double mean = entry.getValue().getMedian();
 if (mean > maxLatency)
 maxLatency = mean;
 }
 // now make another pass to do the weighting based on the maximums we 
found before
-for (Map.Entry entry: 
samples.entrySet())
+for (Map.Entry entry : snapshots.entrySet())
 {
-double score = entry.getValue().getSnapshot().getMedian() / 
maxLatency;
+double score = entry.getValue().getMedian() / maxLatency;
 // finally, add the severity without any weighting, since hosts 
scale this relative to their own load and the size of the task causing the 
severity.
 // "Severity" is basically a measure of compaction activity 
(CASSANDRA-3722).
 if (USE_SEVERITY)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[bdeggleston]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ed0ded12
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ed0ded12
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ed0ded12

Branch: refs/heads/trunk
Commit: ed0ded123c037063f44536dc8363c7985be914a5
Parents: d8f0361 ae78231
Author: Blake Eggleston 
Authored: Tue Dec 5 16:06:29 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 16:06:29 2017 -0800

--
 CHANGES.txt |  2 ++
 .../cassandra/locator/DynamicEndpointSnitch.java| 16 
 2 files changed, 14 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed0ded12/CHANGES.txt
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14090) stress.generate.Distribution.average broken on trunk

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-14090:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

thanks, committed as {{d8f0361228aec146689b8f9402c7aa61d95bdc05}}

> stress.generate.Distribution.average broken on trunk
> 
>
> Key: CASSANDRA-14090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14090
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> Looks like the lgtm.com fixes slightly changed the behavior of 
> Distribution.average, which prevents stress from starting up



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Fix Distribution.average in cassandra-stress

[bdeggleston]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 2402acd47 -> d8f036122


Fix Distribution.average in cassandra-stress

Patch by Blake Eggleston; Reviewed by Jon Haddad for CASSANDRA-14090


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d8f03612
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d8f03612
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d8f03612

Branch: refs/heads/trunk
Commit: d8f0361228aec146689b8f9402c7aa61d95bdc05
Parents: 2402acd
Author: Blake Eggleston 
Authored: Fri Dec 1 14:50:57 2017 -0800
Committer: Blake Eggleston 
Committed: Tue Dec 5 15:58:59 2017 -0800

--
 CHANGES.txt| 1 +
 .../src/org/apache/cassandra/stress/generate/Distribution.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d8f03612/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 51f8961..f3f552c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Fix Distribution.average in cassandra-stress (CASSANDRA-14090)
  * Support a means of logging all queries as they were invoked 
(CASSANDRA-13983)
  * Presize collections (CASSANDRA-13760)
  * Add GroupCommitLogService (CASSANDRA-13530)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d8f03612/tools/stress/src/org/apache/cassandra/stress/generate/Distribution.java
--
diff --git 
a/tools/stress/src/org/apache/cassandra/stress/generate/Distribution.java 
b/tools/stress/src/org/apache/cassandra/stress/generate/Distribution.java
index 016..30efd34 100644
--- a/tools/stress/src/org/apache/cassandra/stress/generate/Distribution.java
+++ b/tools/stress/src/org/apache/cassandra/stress/generate/Distribution.java
@@ -46,7 +46,7 @@ public abstract class Distribution implements Serializable
 {
 double sum = 0;
 double d = 0d;
-for (int count = 0; count < 51 ; count++)
+for (int count = 0; count < 50 ; count++)
 {
 sum += inverseCumProb(d);
 d += 0.02d;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13985) Support restricting reads and writes to specific datacenters on a per user basis

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279342#comment-16279342
 ] 

Blake Eggleston commented on CASSANDRA-13985:
-

Coming back to this, I think that the implicit granting by omission seems like 
the least bad option. Basically wildcard by default. I expect the most common 
use case will be to confine a user to a single data center. So I’m looking at 
this this from the perspective of making that easy to achieve.

I don’t think implementing this as a straight up whitelist would work because 
it creates a chicken and egg problem of turning on the authz feature and being 
able to connect to it and configure it. Also, if you’re only interested in 
confining a subset of all your users, it becomes difficult to administer. On 
the other hand, a black list would add the overhead of having to go and update 
each restricted role any time a dc is added. Obviously it’s not something that 
happens often, but I don’t think we need to add another detail that needs to be 
worried about (and easily overlooked) if we can avoid it.

> Support restricting reads and writes to specific datacenters on a per user 
> basis
> 
>
> Key: CASSANDRA-13985
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13985
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> There are a few use cases where it makes sense to restrict the operations a 
> given user can perform in specific data centers. The obvious use case is the 
> production/analytics datacenter configuration. You don’t want the production 
> user to be reading/or writing to the analytics datacenter, and you don’t want 
> the analytics user to be reading from the production datacenter.
> Although we expect users to get this right on that application level, we 
> should also be able to enforce this at the database level. The first approach 
> that comes to mind would be to support an optional DC parameter when granting 
> select and modify permissions to roles. Something like {{GRANT SELECT ON 
> some_keyspace TO that_user IN DC dc1}}, statements that omit the dc would 
> implicitly be granting permission to all dcs. However, I’m not married to 
> this approach.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14098) Potential Integer Overflow

[songwanging (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

songwanging updated CASSANDRA-14098:

Description: 
Our tool DeepTect has detected a potential integer overflow: 

Path: cassandra/src/java/org/apache/cassandra/service/StorageService.java

{code:java}
...
long totalRowCountEstimate = cfs.estimatedKeysForRange(range);
...
 int splitCount = Math.max(1, Math.min(maxSplitCount, 
(int)(totalRowCountEstimate / keysPerSplit)));
{code}

In the above code snippet, "totalRowCountEstimate" is a long variable, 
"keysPerSplit" is an integer variable. If "totalRowCountEstimate" is super 
large, directly casting "(totalRowCountEstimate / keysPerSplit" into integer 
will definitely lead to a potential integer overflow.

  was:
Our tool DeepTect has detected a potential integer overflow: 

Path: cassandra/src/java/org/apache/cassandra/service/StorageService.java

{code:java}
 int splitCount = Math.max(1, Math.min(maxSplitCount, 
(int)(totalRowCountEstimate / keysPerSplit)));
{code}

In the above code snippet, "totalRowCountEstimate" is a long variable, 
"keysPerSplit" is an integer variable. If "totalRowCountEstimate" is super 
large, directly casting "(totalRowCountEstimate / keysPerSplit" into integer 
will definitely lead to a potential integer overflow.


> Potential Integer Overflow
> --
>
> Key: CASSANDRA-14098
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14098
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: songwanging
>Priority: Minor
>
> Our tool DeepTect has detected a potential integer overflow: 
> Path: cassandra/src/java/org/apache/cassandra/service/StorageService.java
> {code:java}
> ...
> long totalRowCountEstimate = cfs.estimatedKeysForRange(range);
> ...
>  int splitCount = Math.max(1, Math.min(maxSplitCount, 
> (int)(totalRowCountEstimate / keysPerSplit)));
> {code}
> In the above code snippet, "totalRowCountEstimate" is a long variable, 
> "keysPerSplit" is an integer variable. If "totalRowCountEstimate" is super 
> large, directly casting "(totalRowCountEstimate / keysPerSplit" into integer 
> will definitely lead to a potential integer overflow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14098) Potential Integer Overflow

[songwanging (JIRA)]

songwanging created CASSANDRA-14098:
---

 Summary: Potential Integer Overflow
 Key: CASSANDRA-14098
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14098
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: songwanging
Priority: Minor


Our tool DeepTect has detected a potential integer overflow: 

Path: cassandra/src/java/org/apache/cassandra/service/StorageService.java

{code:java}
 int splitCount = Math.max(1, Math.min(maxSplitCount, 
(int)(totalRowCountEstimate / keysPerSplit)));
{code}

In the above code snippet, "totalRowCountEstimate" is a long variable, 
"keysPerSplit" is an integer variable. If "totalRowCountEstimate" is super 
large, directly casting "(totalRowCountEstimate / keysPerSplit" into integer 
will definitely lead to a potential integer overflow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14097) Per-node stream concurrency

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279231#comment-16279231
 ] 

Jason Brown commented on CASSANDRA-14097:
-

CASSANDRA-12229 started down the road of implementing concurrency in streaming. 
What specific things are you thinking about?

> Per-node stream concurrency
> ---
>
> Key: CASSANDRA-14097
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14097
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Eric Evans
>
> Stream sessions with a remote are bound to a single thread, and when 
> compression is in use this thread can be CPU bound, limiting throughput 
> considerably.  When the number of nodes is small (i.e. when the number of 
> concurrent sessions is also low), rebuilds or bootstrap operations can take a 
> very long time.
> Ideally, data could be streamed from any given remote concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14062) Pluggable CommitLog

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279217#comment-16279217
 ] 

Ariel Weisberg commented on CASSANDRA-14062:


My concern is not so much that people use the plugin and don't upgrade when 
upgrading Cassandra, but that data is lost and then we have to track down that 
it was an issue with interactions between Cassandra and the plugin. The 
starting point for everyone is going to be that Cassandra lost data and it's up 
to us to prove otherwise. With software and hardware we probably won't have 
access to.

Not allowing the plugins at all isn't a viable solution so I have three routes 
I think we might take.

1. If you use a commit log plugin and have issues we aren't going to get 
involved at all. We assume it's the plugin. I've seen this approach taken for 
things like your using JNI and getting segfaults from the JVM? Great, go create 
a reproducer that has no JNI or we won't look at it.
2. The plugin is passed the current Cassandra version and won't start if it the 
version string <= the supported version of the plugin and the plugins don't 
cheat. So we would implement this version checking code in OSS, but the version 
string would be compiled into the plugin. At least then we know the plugin and 
it's tests were run against the current C* version. I would expect a commit log 
plugin to be well tested integration test wise, but we have no control over it.
3. We factor this out into a proper tested interface that detects changes in 
contractual behavior. This exposes much less of C* and makes explicit what we 
do expose. So everything in a new package for exposing to plugins. If you are 
compiling a plugin it should only whitelist classes that are in the plugin 
support package and generate a compile time error if anything outside it is 
used.

#3 is not cheap. It could lead to code duplication or encourage us to avoid 
modifying code because then we would have to duplicate it outside the plugin 
support package.

I am in favor of #2 for now since it's simple and puts the onus for making 
things work on plugin developers. The extra work of #3 only makes sense when we 
want to put in the extra effort of stabilizing the interface to make things 
easier for plugin developers. If we can live with plugins being compiled and 
matched to specific C* releases then #2 has the same level of safety and is 
easy to get up and running.

> Pluggable CommitLog
> ---
>
> Key: CASSANDRA-14062
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14062
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Rei Odaira
>Assignee: Rei Odaira
>  Labels: features
> Fix For: 4.x
>
> Attachments: pluggable-commitlog-src.patch, 
> pluggable-commitlog-test.patch
>
>
> This proposal is to make CommitLog pluggable, as discussed in [the Cassandra 
> dev mailing 
> list|https://lists.apache.org/thread.html/1936194d86f5954fa099ced9a0733458eb3249bff3fae3e03e2d1bd8@%3Cdev.cassandra.apache.org%3E].
> We are developing a Cassandra plugin to store CommitLog on our low-latency 
> Flash device (CAPI-Flash). To do that, the original CommitLog interface must 
> be changed to allow plugins. Synching to CommitLog is one of the performance 
> bottlenecks in Cassandra especially with batch commit. I think the pluggable 
> CommitLog will allow other interesting alternatives, such as one using SPDK.
> Our high-level design is similar to the CacheProvider framework
> in org.apache.cassandra.cache:
> * Introduce a new interface, ICommitLog, with methods like 
> getCurrentPosition(), add(), shutdownBlocking(), etc.
> * CommitLog implements ICommitLog.
> * Introduce a new interface, CommitLogProvider, with a create() method, 
> returning ICommitLog.
> * Introduce a new class FileCommitLogProvider implementing CommitLogProvider, 
> to return a singleton instance of CommitLog.
> * Introduce a new property in cassandra.yaml, commitlog_class_name, which 
> specifies what CommitLogProvider to use.  The default is 
> FileCommitLogProvider.
> * Introduce a new class, CommitLogHelper, that loads the class specified by 
> the commitlog_class_name property, creates an instance, and stores it to 
> CommitLogHelper.instance.
> * Replace all of the references to CommitLog.instance with 
> CommitLogHelper.instance.
> Attached are two patches. "pluggable-commitlog-src.patch" is for changes in 
> the src directory, and "pluggable-commitlog-test.patch" is for the test 
> directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14062) Pluggable CommitLog

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279168#comment-16279168
 ] 

Jeff Jirsa commented on CASSANDRA-14062:


I suspect one of the fundamental differences between other pluggable interfaces 
like (row cache, compaction, replication strategies, partitioners), and the 
commitlog, is that the commitlog really MUST interact with an awful lot of the 
internals to be even remotely functional.

If we actively try to allow for third party pluggable commitlogs, one of the 
main concerns seems to be how we guarantee that we don't break your commitlog 
in a minor version - Keyspace, Mutation, PartitionUpdate, SerializationHelper - 
those are all private internal APIs, and they change FREQUENTLY. It seems very 
likely that we'd break your implementation frequently, and potentially in very 
subtle ways that may not be user-obvious. 

I **really** want to find a way to make this work, but I think Ariel's concern 
here is real - if we update PartitionUpdate from 4.0.0 to 4.0.1, your jar for 
4.0.0 may break in a horrible way, and I don't think we can commit to not 
changing those interfaces - the exposure surface is huge, we can't give up that 
much optionality. Depending primarily on CommitLogReplayer as a way to minimize 
that exposure sounds like something worth investigating. 

> Pluggable CommitLog
> ---
>
> Key: CASSANDRA-14062
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14062
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Rei Odaira
>Assignee: Rei Odaira
>  Labels: features
> Fix For: 4.x
>
> Attachments: pluggable-commitlog-src.patch, 
> pluggable-commitlog-test.patch
>
>
> This proposal is to make CommitLog pluggable, as discussed in [the Cassandra 
> dev mailing 
> list|https://lists.apache.org/thread.html/1936194d86f5954fa099ced9a0733458eb3249bff3fae3e03e2d1bd8@%3Cdev.cassandra.apache.org%3E].
> We are developing a Cassandra plugin to store CommitLog on our low-latency 
> Flash device (CAPI-Flash). To do that, the original CommitLog interface must 
> be changed to allow plugins. Synching to CommitLog is one of the performance 
> bottlenecks in Cassandra especially with batch commit. I think the pluggable 
> CommitLog will allow other interesting alternatives, such as one using SPDK.
> Our high-level design is similar to the CacheProvider framework
> in org.apache.cassandra.cache:
> * Introduce a new interface, ICommitLog, with methods like 
> getCurrentPosition(), add(), shutdownBlocking(), etc.
> * CommitLog implements ICommitLog.
> * Introduce a new interface, CommitLogProvider, with a create() method, 
> returning ICommitLog.
> * Introduce a new class FileCommitLogProvider implementing CommitLogProvider, 
> to return a singleton instance of CommitLog.
> * Introduce a new property in cassandra.yaml, commitlog_class_name, which 
> specifies what CommitLogProvider to use.  The default is 
> FileCommitLogProvider.
> * Introduce a new class, CommitLogHelper, that loads the class specified by 
> the commitlog_class_name property, creates an instance, and stores it to 
> CommitLogHelper.instance.
> * Replace all of the references to CommitLog.instance with 
> CommitLogHelper.instance.
> Attached are two patches. "pluggable-commitlog-src.patch" is for changes in 
> the src directory, and "pluggable-commitlog-test.patch" is for the test 
> directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14097) Per-node stream concurrency

[Eric Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-14097:
---
Component/s: Streaming and Messaging

> Per-node stream concurrency
> ---
>
> Key: CASSANDRA-14097
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14097
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Eric Evans
>
> Stream sessions with a remote are bound to a single thread, and when 
> compression is in use this thread can be CPU bound, limiting throughput 
> considerably.  When the number of nodes is small (i.e. when the number of 
> concurrent sessions is also low), rebuilds or bootstrap operations can take a 
> very long time.
> Ideally, data could be streamed from any given remote concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11303) New inbound throughput parameters for streaming

[Eric Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-11303:
---
Issue Type: New Feature  (was: Improvement)

> New inbound throughput parameters for streaming
> ---
>
> Key: CASSANDRA-11303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Configuration
>Reporter: Satoshi Konno
>Assignee: Satoshi Konno
>Priority: Minor
> Attachments: 11303_inbound_limit_debug_20160419.log, 
> 11303_inbound_nolimit_debug_20160419.log, 
> 11303_inbound_patch_for_trunk_20160419.diff, 
> 11303_inbound_patch_for_trunk_20160525.diff, 
> 11303_inbound_patch_for_trunk_20160704.diff, 
> 200vs40inboundstreamthroughput.png, cassandra_inbound_stream.diff
>
>
> Hi,
> To specify stream throughputs of a node more clearly, I would like to add the 
> following new inbound parameters like existing outbound parameters in the 
> cassandra.yaml.
> - stream_throughput_inbound_megabits_per_sec
> - inter_dc_stream_throughput_outbound_megabits_per_sec  
> We use only the existing outbound parameters now, but it is difficult to 
> control the total throughputs of a node.　In our production network, some 
> critical alerts occurs when a node exceed the specified total throughput 
> which is the sum of the input and output throughputs.
> In our operation of Cassandra, the alerts occurs during the bootstrap or 
> repair processing when a new node is added. In the worst case, we have to 
> stop the operation of the exceed node.
> I have attached the patch under consideration. I would like to add a new 
> limiter class, StreamInboundRateLimiter, and use the limiter class in 
> StreamDeserializer class. I use Row::dataSize( ）to get the input throughput 
> in StreamDeserializer::newPartition(), but I am not sure whether the 
> dataSize() returns the correct data size.
> Can someone please tell me how to do it ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11303) New inbound throughput parameters for streaming

[Eric Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-11303:
---
Issue Type: Improvement  (was: New Feature)

> New inbound throughput parameters for streaming
> ---
>
> Key: CASSANDRA-11303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11303
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Satoshi Konno
>Assignee: Satoshi Konno
>Priority: Minor
> Attachments: 11303_inbound_limit_debug_20160419.log, 
> 11303_inbound_nolimit_debug_20160419.log, 
> 11303_inbound_patch_for_trunk_20160419.diff, 
> 11303_inbound_patch_for_trunk_20160525.diff, 
> 11303_inbound_patch_for_trunk_20160704.diff, 
> 200vs40inboundstreamthroughput.png, cassandra_inbound_stream.diff
>
>
> Hi,
> To specify stream throughputs of a node more clearly, I would like to add the 
> following new inbound parameters like existing outbound parameters in the 
> cassandra.yaml.
> - stream_throughput_inbound_megabits_per_sec
> - inter_dc_stream_throughput_outbound_megabits_per_sec  
> We use only the existing outbound parameters now, but it is difficult to 
> control the total throughputs of a node.　In our production network, some 
> critical alerts occurs when a node exceed the specified total throughput 
> which is the sum of the input and output throughputs.
> In our operation of Cassandra, the alerts occurs during the bootstrap or 
> repair processing when a new node is added. In the worst case, we have to 
> stop the operation of the exceed node.
> I have attached the patch under consideration. I would like to add a new 
> limiter class, StreamInboundRateLimiter, and use the limiter class in 
> StreamDeserializer class. I use Row::dataSize( ）to get the input throughput 
> in StreamDeserializer::newPartition(), but I am not sure whether the 
> dataSize() returns the correct data size.
> Can someone please tell me how to do it ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14097) Per-node stream concurrency

[Eric Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-14097:
---
Issue Type: Improvement  (was: Bug)

> Per-node stream concurrency
> ---
>
> Key: CASSANDRA-14097
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14097
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Eric Evans
>
> Stream sessions with a remote are bound to a single thread, and when 
> compression is in use this thread can be CPU bound, limiting throughput 
> considerably.  When the number of nodes is small (i.e. when the number of 
> concurrent sessions is also low), rebuilds or bootstrap operations can take a 
> very long time.
> Ideally, data could be streamed from any given remote concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14097) Per-node stream concurrency

[Eric Evans (JIRA)]

Eric Evans created CASSANDRA-14097:
--

 Summary: Per-node stream concurrency
 Key: CASSANDRA-14097
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14097
 Project: Cassandra
  Issue Type: Bug
Reporter: Eric Evans


Stream sessions with a remote are bound to a single thread, and when 
compression is in use this thread can be CPU bound, limiting throughput 
considerably.  When the number of nodes is small (i.e. when the number of 
concurrent sessions is also low), rebuilds or bootstrap operations can take a 
very long time.

Ideally, data could be streamed from any given remote concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-8523) Writes should be sent to a replacement node which has a new IP while it is streaming in data

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-8523:
---
Summary: Writes should be sent to a replacement node which has a new IP 
while it is streaming in data  (was: Writes should be sent to a replacement 
node while it is streaming in data)

> Writes should be sent to a replacement node which has a new IP while it is 
> streaming in data
> 
>
> Key: CASSANDRA-8523
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8523
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Richard Wagner
>Assignee: Paulo Motta
> Fix For: 2.2.8, 3.0.9, 3.10
>
>
> In our operations, we make heavy use of replace_address (or 
> replace_address_first_boot) in order to replace broken nodes. We now realize 
> that writes are not sent to the replacement nodes while they are in hibernate 
> state and streaming in data. This runs counter to what our expectations were, 
> especially since we know that writes ARE sent to nodes when they are 
> bootstrapped into the ring.
> It seems like cassandra should arrange to send writes to a node that is in 
> the process of replacing another node, just like it does for a nodes that are 
> bootstraping. I hesitate to phrase this as "we should send writes to a node 
> in hibernate" because the concept of hibernate may be useful in other 
> contexts, as per CASSANDRA-8336. Maybe a new state is needed here?
> Among other things, the fact that we don't get writes during this period 
> makes subsequent repairs more expensive, proportional to the number of writes 
> that we miss (and depending on the amount of data that needs to be streamed 
> during replacement and the time it may take to rebuild secondary indexes, we 
> could miss many many hours worth of writes). It also leaves us more exposed 
> to consistency violations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14062) Pluggable CommitLog

[Rei Odaira (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279027#comment-16279027
 ] 

Rei Odaira commented on CASSANDRA-14062:


In the current implementation, our replayer accesses the following Cassandra 
classes:

import org.apache.cassandra.concurrent.Stage;
import org.apache.cassandra.concurrent.StageManager;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.db.commitlog.CommitLogPosition;
import org.apache.cassandra.db.partitions.PartitionUpdate;
import org.apache.cassandra.db.rows.SerializationHelper;
import org.apache.cassandra.io.util.DataInputBuffer;
import org.apache.cassandra.io.util.RebufferingInputStream;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.TableId;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.WrappedRunnable;

I have to admit that so far we have never tried to minimize these dependencies. 
 Probably we can share some logic with 
org.apache.cassandra.db.commitlog.CommitLogReplayer, so that we depend mostly 
on CommitLogReplayer and not much on other classes.
https://github.com/ppc64le/capi-rowcache/blob/capi-commitlog/src/com/ibm/capiflash/cassandra/commitlog/FlashBulkReplayer.java

> Pluggable CommitLog
> ---
>
> Key: CASSANDRA-14062
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14062
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Rei Odaira
>Assignee: Rei Odaira
>  Labels: features
> Fix For: 4.x
>
> Attachments: pluggable-commitlog-src.patch, 
> pluggable-commitlog-test.patch
>
>
> This proposal is to make CommitLog pluggable, as discussed in [the Cassandra 
> dev mailing 
> list|https://lists.apache.org/thread.html/1936194d86f5954fa099ced9a0733458eb3249bff3fae3e03e2d1bd8@%3Cdev.cassandra.apache.org%3E].
> We are developing a Cassandra plugin to store CommitLog on our low-latency 
> Flash device (CAPI-Flash). To do that, the original CommitLog interface must 
> be changed to allow plugins. Synching to CommitLog is one of the performance 
> bottlenecks in Cassandra especially with batch commit. I think the pluggable 
> CommitLog will allow other interesting alternatives, such as one using SPDK.
> Our high-level design is similar to the CacheProvider framework
> in org.apache.cassandra.cache:
> * Introduce a new interface, ICommitLog, with methods like 
> getCurrentPosition(), add(), shutdownBlocking(), etc.
> * CommitLog implements ICommitLog.
> * Introduce a new interface, CommitLogProvider, with a create() method, 
> returning ICommitLog.
> * Introduce a new class FileCommitLogProvider implementing CommitLogProvider, 
> to return a singleton instance of CommitLog.
> * Introduce a new property in cassandra.yaml, commitlog_class_name, which 
> specifies what CommitLogProvider to use.  The default is 
> FileCommitLogProvider.
> * Introduce a new class, CommitLogHelper, that loads the class specified by 
> the commitlog_class_name property, creates an instance, and stores it to 
> CommitLogHelper.instance.
> * Replace all of the references to CommitLog.instance with 
> CommitLogHelper.instance.
> Attached are two patches. "pluggable-commitlog-src.patch" is for changes in 
> the src directory, and "pluggable-commitlog-test.patch" is for the test 
> directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-14091) DynamicSnitch creates a lot of garbage

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14091:
--
Comment: was deleted

(was: {{samples}} can change from where cache is filled, causing an index out 
of bounds exception. This exception isnt caught and will kill the thread in the 
dynamic snitch so they would no longer be computed.)

> DynamicSnitch creates a lot of garbage
> --
>
> Key: CASSANDRA-14091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14091
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The ExponentiallyDecayingReservoir snapshots we take during score updates 
> generate a lot of garbage, and we call getSnapshot twice per endpoint when we 
> only need to call it once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14091) DynamicSnitch creates a lot of garbage

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278948#comment-16278948
 ] 

Chris Lohfink commented on CASSANDRA-14091:
---

{{samples}} can change from where cache is filled, causing an index out of 
bounds exception. This exception isnt caught and will kill the thread in the 
dynamic snitch so they would no longer be computed.

> DynamicSnitch creates a lot of garbage
> --
>
> Key: CASSANDRA-14091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14091
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> The ExponentiallyDecayingReservoir snapshots we take during score updates 
> generate a lot of garbage, and we call getSnapshot twice per endpoint when we 
> only need to call it once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13987) Multithreaded commitlog subtly changed durability

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278517#comment-16278517
 ] 

Jason Brown edited comment on CASSANDRA-13987 at 12/5/17 1:17 PM:
--

committed as sha {{05cb556f90dbd1929a180254809e05620265419b}}

[~beobal] I naively was trying to avoid re-reading the same volatile 
({{shutdown}}), but then looked at it again and I"m reading another volatile 
right after it in the if check ({{syncRequested}}) - further, we'll reread 
{{shutdown}} again if we get a true for the expression and execute the if 
block. Needless to say, my optimization is petty and probably confusing, so 
I've changed it like you correctly recommended.


was (Author: jasobrown):
committed as sha {{05cb556f90dbd1929a180254809e05620265419b}}

> Multithreaded commitlog subtly changed durability
> -
>
> Key: CASSANDRA-13987
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13987
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> When multithreaded commitlog was introduced in CASSANDRA-3578, we subtly 
> changed the way that commitlog durability worked. Everything still gets 
> written to an mmap file. However, not everything is replayable from the 
> mmaped file after a process crash, in periodic mode.
> In brief, the reason this changesd is due to the chained markers that are 
> required for the multithreaded commit log. At each msync, we wait for 
> outstanding mutations to serialize into the commitlog, and update a marker 
> before and after the commits that have accumluated since the last sync. With 
> those markers, we can safely replay that section of the commitlog. Without 
> the markers, we have no guarantee that the commits in that section were 
> successfully written, thus we abandon those commits on replay.
> If you have correlated process failures of multiple nodes at "nearly" the 
> same time (see ["There Is No 
> Now"|http://queue.acm.org/detail.cfm?id=2745385]), it is possible to have 
> data loss if none of the nodes msync the commitlog. For example, with RF=3, 
> if quorum write succeeds on two nodes (and we acknowledge the write back to 
> the client), and then the process on both nodes OOMs (say, due to reading the 
> index for a 100GB partition), the write will be lost if neither process 
> msync'ed the commitlog. More exactly, the commitlog cannot be fully replayed. 
> The reason why this data is silently lost is due to the chained markers that 
> were introduced with CASSANDRA-3578.
> The problem we are addressing with this ticket is incrementally improving 
> 'durability' due to process crash, not host crash. (Note: operators should 
> use batch mode to ensure greater durability, but batch mode in it's current 
> implementation is a) borked, and b) will burn through, *very* rapidly, SSDs 
> that don't have a non-volatile write cache sitting in front.) 
> The current default for {{commitlog_sync_period_in_ms}} is 10 seconds, which 
> means that a node could lose up to ten seconds of data due to process crash. 
> The unfortunate thing is that the data is still avaialble, in the mmap file, 
> but we can't replay it due to incomplete chained markers.
> ftr, I don't believe we've ever had a stated policy about commitlog 
> durability wrt process crash. Pre-2.0 we naturally piggy-backed off the 
> memory mapped file and the fact that every mutation was acquired a lock and 
> wrote into the mmap buffer, and the ability to replay everything out of it 
> came for free. With CASSANDRA-3578, that was subtly changed. 
> Something [~jjirsa] pointed out to me is that [MySQL provides a way to adjust 
> the durability 
> guarantees|https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit]
>  of each commit in innodb via the {{innodb_flush_log_at_trx_commit}}. I'm 
> using that idea as a loose springboard for what to do here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13987) Multithreaded commitlog subtly changed durability

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-13987.
-
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   3.11.2
   3.0.16

committed as sha {{05cb556f90dbd1929a180254809e05620265419b}}

> Multithreaded commitlog subtly changed durability
> -
>
> Key: CASSANDRA-13987
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13987
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> When multithreaded commitlog was introduced in CASSANDRA-3578, we subtly 
> changed the way that commitlog durability worked. Everything still gets 
> written to an mmap file. However, not everything is replayable from the 
> mmaped file after a process crash, in periodic mode.
> In brief, the reason this changesd is due to the chained markers that are 
> required for the multithreaded commit log. At each msync, we wait for 
> outstanding mutations to serialize into the commitlog, and update a marker 
> before and after the commits that have accumluated since the last sync. With 
> those markers, we can safely replay that section of the commitlog. Without 
> the markers, we have no guarantee that the commits in that section were 
> successfully written, thus we abandon those commits on replay.
> If you have correlated process failures of multiple nodes at "nearly" the 
> same time (see ["There Is No 
> Now"|http://queue.acm.org/detail.cfm?id=2745385]), it is possible to have 
> data loss if none of the nodes msync the commitlog. For example, with RF=3, 
> if quorum write succeeds on two nodes (and we acknowledge the write back to 
> the client), and then the process on both nodes OOMs (say, due to reading the 
> index for a 100GB partition), the write will be lost if neither process 
> msync'ed the commitlog. More exactly, the commitlog cannot be fully replayed. 
> The reason why this data is silently lost is due to the chained markers that 
> were introduced with CASSANDRA-3578.
> The problem we are addressing with this ticket is incrementally improving 
> 'durability' due to process crash, not host crash. (Note: operators should 
> use batch mode to ensure greater durability, but batch mode in it's current 
> implementation is a) borked, and b) will burn through, *very* rapidly, SSDs 
> that don't have a non-volatile write cache sitting in front.) 
> The current default for {{commitlog_sync_period_in_ms}} is 10 seconds, which 
> means that a node could lose up to ten seconds of data due to process crash. 
> The unfortunate thing is that the data is still avaialble, in the mmap file, 
> but we can't replay it due to incomplete chained markers.
> ftr, I don't believe we've ever had a stated policy about commitlog 
> durability wrt process crash. Pre-2.0 we naturally piggy-backed off the 
> memory mapped file and the fact that every mutation was acquired a lock and 
> wrote into the mmap buffer, and the ability to replay everything out of it 
> came for free. With CASSANDRA-3578, that was subtly changed. 
> Something [~jjirsa] pointed out to me is that [MySQL provides a way to adjust 
> the durability 
> guarantees|https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit]
>  of each commit in innodb via the {{innodb_flush_log_at_trx_commit}}. I'm 
> using that idea as a loose springboard for what to do here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2402acd4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2402acd4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2402acd4

Branch: refs/heads/trunk
Commit: 2402acd47e3bb514981cde742b7330666c564869
Parents: d274c6a c3a1a4f
Author: Jason Brown 
Authored: Tue Dec 5 05:10:08 2017 -0800
Committer: Jason Brown 
Committed: Tue Dec 5 05:11:21 2017 -0800

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 10 +-
 .../org/apache/cassandra/config/Config.java |  1 +
 .../cassandra/config/DatabaseDescriptor.java| 10 ++
 .../AbstractCommitLogSegmentManager.java|  8 +-
 .../db/commitlog/AbstractCommitLogService.java  | 84 ++---
 .../cassandra/db/commitlog/CommitLog.java   |  4 +-
 .../db/commitlog/CommitLogSegment.java  | 90 --
 .../db/commitlog/CompressedSegment.java |  4 +-
 .../db/commitlog/EncryptedSegment.java  |  3 -
 .../db/commitlog/FileDirectSegment.java | 14 +++
 .../db/commitlog/MemoryMappedSegment.java   |  4 +
 .../db/commitlog/PeriodicCommitLogService.java  |  2 +-
 .../cassandra/db/commitlog/CDCTestReplayer.java |  2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 98 
 .../CommitLogSegmentBackpressureTest.java   |  4 +-
 .../CommitLogSegmentManagerCDCTest.java |  8 +-
 .../cassandra/db/commitlog/CommitLogTest.java   | 10 +-
 .../db/commitlog/CommitLogTestReplayer.java |  2 +-
 19 files changed, 288 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2402acd4/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2402acd4/conf/cassandra.yaml
--
diff --cc conf/cassandra.yaml
index ba478e7,3569d36..02cec12
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@@ -358,24 -367,21 +358,24 @@@ counter_cache_save_period: 720
  # If not set, the default directory is $CASSANDRA_HOME/data/saved_caches.
  # saved_caches_directory: /var/lib/cassandra/saved_caches
  
 -# commitlog_sync may be either "periodic" or "batch." 
 +# commitlog_sync may be either "periodic", "group", or "batch." 
  # 
  # When in batch mode, Cassandra won't ack writes until the commit log
 -# has been fsynced to disk.  It will wait
 -# commitlog_sync_batch_window_in_ms milliseconds between fsyncs.
 -# This window should be kept short because the writer threads will
 -# be unable to do extra work while waiting.  (You may need to increase
 -# concurrent_writes for the same reason.)
 +# has been flushed to disk.  Each incoming write will trigger the flush task.
 +# commitlog_sync_batch_window_in_ms is a deprecated value. Previously it had
 +# almost no value, and is being removed.
  #
 -# commitlog_sync: batch
  # commitlog_sync_batch_window_in_ms: 2
  #
 -# the other option is "periodic" where writes may be acked immediately
 +# group mode is similar to batch mode, where Cassandra will not ack writes
 +# until the commit log has been flushed to disk. The difference is group
 +# mode will wait up to commitlog_sync_group_window_in_ms between flushes.
 +#
 +# commitlog_sync_group_window_in_ms: 1000
 +#
 +# the default option is "periodic" where writes may be acked immediately
  # and the CommitLog is simply synced every commitlog_sync_period_in_ms
- # milliseconds. 
+ # milliseconds.
  commitlog_sync: periodic
  commitlog_sync_period_in_ms: 1
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2402acd4/src/java/org/apache/cassandra/config/Config.java
--
diff --cc src/java/org/apache/cassandra/config/Config.java
index 1db8217,5fe752e..4fa3bed
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -185,13 -198,9 +185,14 @@@ public class Confi
  public String commitlog_directory;
  public Integer commitlog_total_space_in_mb;
  public CommitLogSync commitlog_sync;
 +
 +/**
 + * @deprecated since 4.0 This value was near useless, and we're not using 
it anymore
 + */
  public double commitlog_sync_batch_window_in_ms = Double.NaN;
 +public double commitlog_sync_group_window_in_ms = Double.NaN;
  public int commitlog_sync_period_in_ms;
+ public int commitlog_marker_period_in_ms = 0;
  public int commitlog_segment_size_in_mb = 32;
  public ParameterizedClass commitlog_compression;
  public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cass

[2/6] cassandra git commit: More frequent commitlog chained markers

[jasobrown]

More frequent commitlog chained markers

patch by jasobrown; reviewed by Sam Tunnecliffe for CASSANDRA-13987


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/05cb556f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/05cb556f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/05cb556f

Branch: refs/heads/cassandra-3.11
Commit: 05cb556f90dbd1929a180254809e05620265419b
Parents: eb05025
Author: Jason Brown 
Authored: Tue Oct 17 15:37:13 2017 -0700
Committer: Jason Brown 
Committed: Tue Dec 5 05:06:38 2017 -0800

--
 CHANGES.txt |   1 +
 conf/cassandra.yaml |  10 +-
 .../org/apache/cassandra/config/Config.java |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  10 ++
 .../db/commitlog/AbstractCommitLogService.java  |  86 ++---
 .../cassandra/db/commitlog/CommitLog.java   |   4 +-
 .../db/commitlog/CommitLogSegment.java  |  85 
 .../db/commitlog/CompressedSegment.java |  12 ++
 .../db/commitlog/MemoryMappedSegment.java   |   7 +-
 .../db/commitlog/PeriodicCommitLogService.java  |   2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 128 +++
 .../CommitLogSegmentBackpressureTest.java   |   4 +-
 .../cassandra/db/commitlog/CommitLogTest.java   |   6 +-
 .../db/commitlog/CommitLogTestReplayer.java |   2 +-
 14 files changed, 304 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c50a3f..2683dc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index b783090..71e0b2a 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -300,10 +300,18 @@ counter_cache_save_period: 7200
 #
 # the other option is "periodic" where writes may be acked immediately
 # and the CommitLog is simply synced every commitlog_sync_period_in_ms
-# milliseconds. 
+# milliseconds.
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 1
 
+# Time interval in millis at which we should update the chained markers in the 
commitlog.
+# This allows more of the commitlog to be replayed from the mmapped file
+# if the cassandra process crashes; this does not help in durability for 
surviving a host fail.
+# This value only makes sense if it is significantly less that 
commitlog_sync_period_in_ms,
+# and only applies to periodic mode when not using commitlog compression or 
encryption.
+# commitlog_marker_period_in_ms: 100
+
+
 # The size of the individual commitlog file segments.  A commitlog
 # segment may be archived, deleted, or recycled once all the data
 # in it (potentially from each columnfamily in the system) has been

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 64d41bb..0796183 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -192,6 +192,7 @@ public class Config
 public CommitLogSync commitlog_sync;
 public Double commitlog_sync_batch_window_in_ms;
 public Integer commitlog_sync_period_in_ms;
+public Integer commitlog_marker_period_in_ms = 0;
 public int commitlog_segment_size_in_mb = 32;
 public ParameterizedClass commitlog_compression;
 public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index efc71ef..169ed3d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -1519,6 +1519,16 @@ public class DatabaseDescriptor
 conf.commitlog_sync_period_in_ms = periodMillis;
 }
 
+public static void setCommitLogMarkerPeriod(int marke

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c3a1a4fa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c3a1a4fa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c3a1a4fa

Branch: refs/heads/cassandra-3.11
Commit: c3a1a4fa8083b2c4c6a5454551979954a0a71339
Parents: d577918 05cb556
Author: Jason Brown 
Authored: Tue Dec 5 05:07:18 2017 -0800
Committer: Jason Brown 
Committed: Tue Dec 5 05:08:22 2017 -0800

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 10 +-
 .../org/apache/cassandra/config/Config.java |  1 +
 .../cassandra/config/DatabaseDescriptor.java| 10 ++
 .../AbstractCommitLogSegmentManager.java|  8 +-
 .../db/commitlog/AbstractCommitLogService.java  | 84 ++---
 .../cassandra/db/commitlog/CommitLog.java   |  4 +-
 .../db/commitlog/CommitLogSegment.java  | 85 +++--
 .../db/commitlog/CompressedSegment.java |  4 +-
 .../db/commitlog/EncryptedSegment.java  |  3 -
 .../db/commitlog/FileDirectSegment.java | 14 +++
 .../db/commitlog/MemoryMappedSegment.java   |  4 +
 .../db/commitlog/PeriodicCommitLogService.java  |  2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 98 
 .../CommitLogSegmentBackpressureTest.java   |  4 +-
 .../cassandra/db/commitlog/CommitLogTest.java   | 10 +-
 .../db/commitlog/CommitLogTestReplayer.java |  2 +-
 17 files changed, 281 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/CHANGES.txt
--
diff --cc CHANGES.txt
index e26aeb8,2683dc2..7c0af91
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,12 -1,5 +1,13 @@@
 -3.0.16
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/config/Config.java
--
diff --cc src/java/org/apache/cassandra/config/Config.java
index a01203c,0796183..5fe752e
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -198,8 -190,9 +198,9 @@@ public class Confi
  public String commitlog_directory;
  public Integer commitlog_total_space_in_mb;
  public CommitLogSync commitlog_sync;
 -public Double commitlog_sync_batch_window_in_ms;
 -public Integer commitlog_sync_period_in_ms;
 -public Integer commitlog_marker_period_in_ms = 0;
 +public double commitlog_sync_batch_window_in_ms = Double.NaN;
 +public int commitlog_sync_period_in_ms;
++public int commitlog_marker_period_in_ms = 0;
  public int commitlog_segment_size_in_mb = 32;
  public ParameterizedClass commitlog_compression;
  public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
index 7f3c9f8,000..2c324aa
mode 100755,00..100755
--- 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
+++ 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
@@@ -1,550 -1,0 +1,552 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF lic

[1/6] cassandra git commit: More frequent commitlog chained markers

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 eb05025c0 -> 05cb556f9
  refs/heads/cassandra-3.11 d577918ba -> c3a1a4fa8
  refs/heads/trunk d274c6ac0 -> 2402acd47


More frequent commitlog chained markers

patch by jasobrown; reviewed by Sam Tunnecliffe for CASSANDRA-13987


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/05cb556f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/05cb556f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/05cb556f

Branch: refs/heads/cassandra-3.0
Commit: 05cb556f90dbd1929a180254809e05620265419b
Parents: eb05025
Author: Jason Brown 
Authored: Tue Oct 17 15:37:13 2017 -0700
Committer: Jason Brown 
Committed: Tue Dec 5 05:06:38 2017 -0800

--
 CHANGES.txt |   1 +
 conf/cassandra.yaml |  10 +-
 .../org/apache/cassandra/config/Config.java |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  10 ++
 .../db/commitlog/AbstractCommitLogService.java  |  86 ++---
 .../cassandra/db/commitlog/CommitLog.java   |   4 +-
 .../db/commitlog/CommitLogSegment.java  |  85 
 .../db/commitlog/CompressedSegment.java |  12 ++
 .../db/commitlog/MemoryMappedSegment.java   |   7 +-
 .../db/commitlog/PeriodicCommitLogService.java  |   2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 128 +++
 .../CommitLogSegmentBackpressureTest.java   |   4 +-
 .../cassandra/db/commitlog/CommitLogTest.java   |   6 +-
 .../db/commitlog/CommitLogTestReplayer.java |   2 +-
 14 files changed, 304 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c50a3f..2683dc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index b783090..71e0b2a 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -300,10 +300,18 @@ counter_cache_save_period: 7200
 #
 # the other option is "periodic" where writes may be acked immediately
 # and the CommitLog is simply synced every commitlog_sync_period_in_ms
-# milliseconds. 
+# milliseconds.
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 1
 
+# Time interval in millis at which we should update the chained markers in the 
commitlog.
+# This allows more of the commitlog to be replayed from the mmapped file
+# if the cassandra process crashes; this does not help in durability for 
surviving a host fail.
+# This value only makes sense if it is significantly less that 
commitlog_sync_period_in_ms,
+# and only applies to periodic mode when not using commitlog compression or 
encryption.
+# commitlog_marker_period_in_ms: 100
+
+
 # The size of the individual commitlog file segments.  A commitlog
 # segment may be archived, deleted, or recycled once all the data
 # in it (potentially from each columnfamily in the system) has been

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 64d41bb..0796183 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -192,6 +192,7 @@ public class Config
 public CommitLogSync commitlog_sync;
 public Double commitlog_sync_batch_window_in_ms;
 public Integer commitlog_sync_period_in_ms;
+public Integer commitlog_marker_period_in_ms = 0;
 public int commitlog_segment_size_in_mb = 32;
 public ParameterizedClass commitlog_compression;
 public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index efc71ef..169ed3d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.j

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c3a1a4fa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c3a1a4fa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c3a1a4fa

Branch: refs/heads/trunk
Commit: c3a1a4fa8083b2c4c6a5454551979954a0a71339
Parents: d577918 05cb556
Author: Jason Brown 
Authored: Tue Dec 5 05:07:18 2017 -0800
Committer: Jason Brown 
Committed: Tue Dec 5 05:08:22 2017 -0800

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 10 +-
 .../org/apache/cassandra/config/Config.java |  1 +
 .../cassandra/config/DatabaseDescriptor.java| 10 ++
 .../AbstractCommitLogSegmentManager.java|  8 +-
 .../db/commitlog/AbstractCommitLogService.java  | 84 ++---
 .../cassandra/db/commitlog/CommitLog.java   |  4 +-
 .../db/commitlog/CommitLogSegment.java  | 85 +++--
 .../db/commitlog/CompressedSegment.java |  4 +-
 .../db/commitlog/EncryptedSegment.java  |  3 -
 .../db/commitlog/FileDirectSegment.java | 14 +++
 .../db/commitlog/MemoryMappedSegment.java   |  4 +
 .../db/commitlog/PeriodicCommitLogService.java  |  2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 98 
 .../CommitLogSegmentBackpressureTest.java   |  4 +-
 .../cassandra/db/commitlog/CommitLogTest.java   | 10 +-
 .../db/commitlog/CommitLogTestReplayer.java |  2 +-
 17 files changed, 281 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/CHANGES.txt
--
diff --cc CHANGES.txt
index e26aeb8,2683dc2..7c0af91
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,12 -1,5 +1,13 @@@
 -3.0.16
 +3.11.2
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * More frequent commitlog chained markers (CASSANDRA-13987)
   * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/config/Config.java
--
diff --cc src/java/org/apache/cassandra/config/Config.java
index a01203c,0796183..5fe752e
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -198,8 -190,9 +198,9 @@@ public class Confi
  public String commitlog_directory;
  public Integer commitlog_total_space_in_mb;
  public CommitLogSync commitlog_sync;
 -public Double commitlog_sync_batch_window_in_ms;
 -public Integer commitlog_sync_period_in_ms;
 -public Integer commitlog_marker_period_in_ms = 0;
 +public double commitlog_sync_batch_window_in_ms = Double.NaN;
 +public int commitlog_sync_period_in_ms;
++public int commitlog_marker_period_in_ms = 0;
  public int commitlog_segment_size_in_mb = 32;
  public ParameterizedClass commitlog_compression;
  public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3a1a4fa/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
index 7f3c9f8,000..2c324aa
mode 100755,00..100755
--- 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
+++ 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
@@@ -1,550 -1,0 +1,552 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses thi

[jira] [Updated] (CASSANDRA-6246) EPaxos

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-6246:

Reviewer:   (was: Sylvain Lebresne)

> EPaxos
> --
>
> Key: CASSANDRA-6246
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6246
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jonathan Ellis
>Assignee: Blake Eggleston
>  Labels: messaging-service-bump-required
> Fix For: 4.x
>
>
> One reason we haven't optimized our Paxos implementation with Multi-paxos is 
> that Multi-paxos requires leader election and hence, a period of 
> unavailability when the leader dies.
> EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, 
> (2) is particularly useful across multiple datacenters, and (3) allows any 
> node to act as coordinator: 
> http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf
> However, there is substantial additional complexity involved if we choose to 
> implement it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: More frequent commitlog chained markers

[jasobrown]

More frequent commitlog chained markers

patch by jasobrown; reviewed by Sam Tunnecliffe for CASSANDRA-13987


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/05cb556f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/05cb556f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/05cb556f

Branch: refs/heads/trunk
Commit: 05cb556f90dbd1929a180254809e05620265419b
Parents: eb05025
Author: Jason Brown 
Authored: Tue Oct 17 15:37:13 2017 -0700
Committer: Jason Brown 
Committed: Tue Dec 5 05:06:38 2017 -0800

--
 CHANGES.txt |   1 +
 conf/cassandra.yaml |  10 +-
 .../org/apache/cassandra/config/Config.java |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  10 ++
 .../db/commitlog/AbstractCommitLogService.java  |  86 ++---
 .../cassandra/db/commitlog/CommitLog.java   |   4 +-
 .../db/commitlog/CommitLogSegment.java  |  85 
 .../db/commitlog/CompressedSegment.java |  12 ++
 .../db/commitlog/MemoryMappedSegment.java   |   7 +-
 .../db/commitlog/PeriodicCommitLogService.java  |   2 +-
 .../commitlog/CommitLogChainedMarkersTest.java  | 128 +++
 .../CommitLogSegmentBackpressureTest.java   |   4 +-
 .../cassandra/db/commitlog/CommitLogTest.java   |   6 +-
 .../db/commitlog/CommitLogTestReplayer.java |   2 +-
 14 files changed, 304 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c50a3f..2683dc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * More frequent commitlog chained markers (CASSANDRA-13987)
  * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index b783090..71e0b2a 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -300,10 +300,18 @@ counter_cache_save_period: 7200
 #
 # the other option is "periodic" where writes may be acked immediately
 # and the CommitLog is simply synced every commitlog_sync_period_in_ms
-# milliseconds. 
+# milliseconds.
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 1
 
+# Time interval in millis at which we should update the chained markers in the 
commitlog.
+# This allows more of the commitlog to be replayed from the mmapped file
+# if the cassandra process crashes; this does not help in durability for 
surviving a host fail.
+# This value only makes sense if it is significantly less that 
commitlog_sync_period_in_ms,
+# and only applies to periodic mode when not using commitlog compression or 
encryption.
+# commitlog_marker_period_in_ms: 100
+
+
 # The size of the individual commitlog file segments.  A commitlog
 # segment may be archived, deleted, or recycled once all the data
 # in it (potentially from each columnfamily in the system) has been

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 64d41bb..0796183 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -192,6 +192,7 @@ public class Config
 public CommitLogSync commitlog_sync;
 public Double commitlog_sync_batch_window_in_ms;
 public Integer commitlog_sync_period_in_ms;
+public Integer commitlog_marker_period_in_ms = 0;
 public int commitlog_segment_size_in_mb = 32;
 public ParameterizedClass commitlog_compression;
 public int commitlog_max_compression_buffers_in_pool = 3;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/05cb556f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index efc71ef..169ed3d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -1519,6 +1519,16 @@ public class DatabaseDescriptor
 conf.commitlog_sync_period_in_ms = periodMillis;
 }
 
+public static void setCommitLogMarkerPeriod(int markerPeriod)

[jira] [Resolved] (CASSANDRA-8994) Make keyspace handling in FunctionName less fragile

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne resolved CASSANDRA-8994.
-
Resolution: Won't Fix

Closing as that was me nitpicking on code in the first place and it probably 
doesn't even apply anymore.

> Make keyspace handling in FunctionName less fragile
> ---
>
> Key: CASSANDRA-8994
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8994
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>Priority: Minor
> Fix For: 4.x
>
>
> {{FunctionName}} keyspace field can be null because it can be omitted by the 
> user (in which case we search both the system keyspace and the one of the 
> statement). The handling of that is imo a tad fragile: the {{equals}} method 
> should typically probably complain if one of its operand has no keyspace 
> since in that case, we can't really properly answer. The code currently work 
> around that in {{Functions}} by avoiding {{equals}} when this matter but it 
> would still be pretty easy to get that wrong, especially since 
> {{FunctionName}} is used in maps.  For instance, you could argue that 
> {{Functions#find}} is broken if its argument has no keyspace (it happens to 
> be only use when that's not the case, but again, pretty easy to get wrong in 
> the future).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14080) Handling 0 size hint files during start

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278491#comment-16278491
 ] 

Aleksey Yeschenko commented on CASSANDRA-14080:
---

If that's the intent, then maybe we should probably generalize it beyond 
handling empty files? As that's just one of many possible manifestations of 
corruption.

> Handling 0 size hint files during start
> ---
>
> Key: CASSANDRA-14080
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14080
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints
>Reporter: Aleksandr Ivanov
>Assignee: Alex Lourie
>
> Continuation of CASSANDRA-12728 bug.
> Problem: Cassandra didn't start due to 0 size hints files
> Log form v3.0.14:
> {code:java}
> INFO  [main] 2017-11-28 19:10:13,554 StorageService.java:575 - Cassandra 
> version: 3.0.14
> INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:576 - Thrift API 
> version: 20.1.0
> INFO  [main] 2017-11-28 19:10:13,555 StorageService.java:577 - CQL supported 
> versions: 3.4.0 (default: 3.4.0)
> ERROR [main] 2017-11-28 19:10:13,592 CassandraDaemon.java:710 - Exception 
> encountered during startup
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) 
> ~[na:1.8.0_141]
> at java.util.Iterator.forEachRemaining(Iterator.java:116) 
> ~[na:1.8.0_141]
> at 
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
>  ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 
> ~[na:1.8.0_141]
> at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) 
> ~[na:1.8.0_141]
> at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsService.(HintsService.java:88) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsService.(HintsService.java:63) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.StorageProxy.(StorageProxy.java:121) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141]
> at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:570)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) 
> [apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
>  [apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) 
> [apache-cassandra-3.0.14.jar:3.0.14]
> Caused by: java.io.EOFException: null
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) 
> ~[na:1.8.0_141]
> at 
> org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> ... 20 common frames omitted
> {code}
> After several 0 size hints files deletion Cassandra started successfully.
> Jeff Jirsa added a comment - Yesterday
> Aleksandr Ivanov can you open a new JIRA and link it back to this one? It's 
> possible that the original patch didn't consider 0 byte files (I don't have 
> time to go back and look at the commit, and it was long enough ago that I've 
> forgotten) - were all of your files 0 bytes?
> Not all, 8..10 hints files were with 0 size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-6590) Gossip does not heal after a temporary partition at startup

[Yap Sok Ann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278454#comment-16278454
 ] 

Yap Sok Ann commented on CASSANDRA-6590:


We just saw this in a cluster running 2.1.19. Maybe the behavior as described 
is not fixed yet?

> Gossip does not heal after a temporary partition at startup
> ---
>
> Key: CASSANDRA-6590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6590
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Brandon Williams
>Assignee: Vijay
> Fix For: 2.0.11
>
> Attachments: 0001-CASSANDRA-6590.patch, 0001-logging-for-6590.patch, 
> 6590_disable_echo.txt
>
>
> See CASSANDRA-6571 for background.  If a node is partitioned on startup when 
> the echo command is sent, but then the partition heals, the halves of the 
> partition will never mark each other up despite being able to communicate.  
> This stems from CASSANDRA-3533.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13987) Multithreaded commitlog subtly changed durability

[Sam Tunnicliffe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278437#comment-16278437
 ] 

Sam Tunnicliffe commented on CASSANDRA-13987:
-

bq. The original code did this (across all three branches)...
Thanks [~jasobrown], I was having a brain fart and overlooking the fact that 
{{lastMarkerOffset}} is of course the start of the marker and so we have 
necessarily allocated past it to write the marker.

I'm +1 on this, once CI is happy (as happy as it can be). I just have one 
niggle, which I'll leave to you to decide whether to address or not...
On 3.0, in the 'should we flush or just update markers' condition, I think it 
would be clearer to check {{shutdown}}, rather than {{!run}} given that {{run = 
!shutdown}}.  



> Multithreaded commitlog subtly changed durability
> -
>
> Key: CASSANDRA-13987
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13987
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> When multithreaded commitlog was introduced in CASSANDRA-3578, we subtly 
> changed the way that commitlog durability worked. Everything still gets 
> written to an mmap file. However, not everything is replayable from the 
> mmaped file after a process crash, in periodic mode.
> In brief, the reason this changesd is due to the chained markers that are 
> required for the multithreaded commit log. At each msync, we wait for 
> outstanding mutations to serialize into the commitlog, and update a marker 
> before and after the commits that have accumluated since the last sync. With 
> those markers, we can safely replay that section of the commitlog. Without 
> the markers, we have no guarantee that the commits in that section were 
> successfully written, thus we abandon those commits on replay.
> If you have correlated process failures of multiple nodes at "nearly" the 
> same time (see ["There Is No 
> Now"|http://queue.acm.org/detail.cfm?id=2745385]), it is possible to have 
> data loss if none of the nodes msync the commitlog. For example, with RF=3, 
> if quorum write succeeds on two nodes (and we acknowledge the write back to 
> the client), and then the process on both nodes OOMs (say, due to reading the 
> index for a 100GB partition), the write will be lost if neither process 
> msync'ed the commitlog. More exactly, the commitlog cannot be fully replayed. 
> The reason why this data is silently lost is due to the chained markers that 
> were introduced with CASSANDRA-3578.
> The problem we are addressing with this ticket is incrementally improving 
> 'durability' due to process crash, not host crash. (Note: operators should 
> use batch mode to ensure greater durability, but batch mode in it's current 
> implementation is a) borked, and b) will burn through, *very* rapidly, SSDs 
> that don't have a non-volatile write cache sitting in front.) 
> The current default for {{commitlog_sync_period_in_ms}} is 10 seconds, which 
> means that a node could lose up to ten seconds of data due to process crash. 
> The unfortunate thing is that the data is still avaialble, in the mmap file, 
> but we can't replay it due to incomplete chained markers.
> ftr, I don't believe we've ever had a stated policy about commitlog 
> durability wrt process crash. Pre-2.0 we naturally piggy-backed off the 
> memory mapped file and the fact that every mutation was acquired a lock and 
> wrote into the mmap buffer, and the ability to replay everything out of it 
> came for free. With CASSANDRA-3578, that was subtly changed. 
> Something [~jjirsa] pointed out to me is that [MySQL provides a way to adjust 
> the durability 
> guarantees|https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit]
>  of each commit in innodb via the {{innodb_flush_log_at_trx_commit}}. I'm 
> using that idea as a loose springboard for what to do here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11409) set read repair chance to 0 but find read repair process in trace

[Hong Pai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278432#comment-16278432
 ] 

Hong Pai commented on CASSANDRA-11409:
--

 As  [~cam1982] said, I found it too.
I use TWCS and set the dclocal_read_repair_chance / read_repair_chance to 0.0 
for a table.

But after I restart my C* nodes on by one without manual nodetool repair , and 
new write and read request come in... I found SSTable overlapping.
I think the only reason is read repair.
See https://issues.apache.org/jira/browse/CASSANDRA-13910

> set read repair chance to 0 but find read repair process in trace
> -
>
> Key: CASSANDRA-11409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11409
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL, Distributed Metadata
> Environment: Cassandra 2.1.13 with centos 7
>Reporter: Ryan Cho
>Priority: Minor
>  Labels: lhf
> Attachments: 螢幕快照 2016-03-23 下午2.06.10.png
>
>
> I have set  dclocal_read_repair_chance and read_repair_chance to 0.0 for one 
> month, but I still find "Read-repair DC_LOCAL" and "Initiating read-repair" 
> activities in system_trace.events, and  query was executed in these two days 
> and long time after set read repair chance to 0.0 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14094) Avoid pointless calls to ThreadLocalRandom

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14094:

Reviewer: Jason Brown

> Avoid pointless calls to ThreadLocalRandom
> --
>
> Key: CASSANDRA-14094
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14094
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> In the compression paths, we probabilistically validate the checksum. In 
> cases where the chance is 100%, we don’t need to call {{ThreadLocalRandom}} .



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14094) Avoid pointless calls to ThreadLocalRandom

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278314#comment-16278314
 ] 

Jason Brown commented on CASSANDRA-14094:
-

+1

> Avoid pointless calls to ThreadLocalRandom
> --
>
> Key: CASSANDRA-14094
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14094
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> In the compression paths, we probabilistically validate the checksum. In 
> cases where the chance is 100%, we don’t need to call {{ThreadLocalRandom}} .



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14094) Avoid pointless calls to ThreadLocalRandom

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14094:

Status: Ready to Commit  (was: Patch Available)

> Avoid pointless calls to ThreadLocalRandom
> --
>
> Key: CASSANDRA-14094
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14094
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> In the compression paths, we probabilistically validate the checksum. In 
> cases where the chance is 100%, we don’t need to call {{ThreadLocalRandom}} .



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14071) Materialized view on table with TTL issue

[Silviu Butnariu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278261#comment-16278261
 ] 

Silviu Butnariu commented on CASSANDRA-14071:
-

Great news that you find and are on your way of fixing the issue! Any idea on 
what version will this fix be launched?

> Materialized view on table with TTL issue
> -
>
> Key: CASSANDRA-14071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14071
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination, Materialized Views
> Environment: Cassandra 3
>Reporter: Silviu Butnariu
>Assignee: ZhaoYang
>  Labels: correctness
>
> Materialized views that cluster by a column that is not part of table's PK 
> and are created from tables that have *default_time_to_live* seems to 
> malfunction.
> Having this table
> {code:java}
> CREATE TABLE sbutnariu.test_bug (
> field1 smallint,
> field2 smallint,
> date timestamp,
> PRIMARY KEY ((field1), field2)
> ) WITH default_time_to_live = 1000;
> {code}
> and the materialized view
> {code:java}
> CREATE MATERIALIZED VIEW sbutnariu.test_bug_by_date AS SELECT * FROM 
> sbutnariu.test_bug WHERE field1 IS NOT NULL AND field2 IS NOT NULL AND date 
> IS NOT NULL PRIMARY KEY ((field1), date, field2) WITH CLUSTERING ORDER BY 
> (date desc, field2 asc);
> {code}
> After inserting 3 rows with same PK (should upsert), the materialized view 
> will have 3 rows.
> {code:java}
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> insert into sbutnariu.test_bug(field1, field2, date) values (1, 2, 
> toTimestamp(now()));
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*3 rows*/
> {code}
> If I remove the ttl and try again, it works as expected:
> {code:java}
> truncate sbutnariu.test_bug;
> alter table sbutnariu.test_bug with default_time_to_live = 0;
> select * from sbutnariu.test_bug; /*1 row*/
> select * from sbutnariu.test_bug_by_date;/*1 row*/
> {code}
> I've tested on versions 3.0.14 and 3.0.15. The bug was introduced in 3.0.15, 
> as in 3.0.14 it works as expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org