[jira] [Commented] (CASSANDRA-13952) C* still sending WriteFailure instead of CDCWriteFailure

2017-10-20 Thread Joshua McKenzie (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213590#comment-16213590
 ] 

Joshua McKenzie commented on CASSANDRA-13952:
-

[~jjirsa]: I'm about 95% sure the answer is yes, working as intended, not a 
bug. The ErrorMessage passing code isn't particularly documented (or I couldn't 
find it?) so I had to infer a lot of its structure and intent, and its 
something I hadn't touched before CASSANDRA-12148 so I could be missing 
something.

This is an error that can happen on any replica while attempting to write to a 
CDC-enabled table when the on-disk footprint hits the limit. Previously I'd 
incorrectly used WriteTimeoutException with the initial implementation in 
CASSANDRA-8844 (no clue how/why I did that...), so changing that to either a 
WriteFailureException or, preferably, defining a new ExceptionCode and having a 
specific CDC_WRITE_FAILURE / CDCWriteException smelled much cleaner.

> C* still sending WriteFailure instead of CDCWriteFailure
> 
>
> Key: CASSANDRA-13952
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13952
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
> Fix For: 4.0
>
>
> I've tested this setting setting cdc_total_space_in_mb: 1 and  with the 
> current python driver master
> {code}
> from cassandra.cluster import Cluster
> cluster = Cluster(protocol_version=5, allow_beta_protocol_version=True)
> session = cluster.connect()
> session.execute("""
> CREATE KEYSPACE IF NOT EXISTS %s
> WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '2' 
> }
> """ % "query_per_range")
> session.execute("""
> CREATE TABLE IF NOT EXISTS query_per_range.mytable_int (
> key_one int,
> key_two int,
> col1 text,
> col2 text,
> PRIMARY KEY ((key_one, key_two), col1)
> )  WITH cdc=true;
> """)
> for i in range(1000):
> session.execute("INSERT INTO query_per_range.mytable_int(key_one, 
> key_two, col1, col2) VALUES (%s, %s, %s, %s)",
> (i, i, str(i), str(i)))
> {code}
> I'm receiving a {{WriteFailure}} every time but I was expecting a 
> {{CDCWriteFailure}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13000) slow query log analysis tool

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213520#comment-16213520
 ] 

Jeff Jirsa commented on CASSANDRA-13000:


[~rustyrazorblade] - as the reporter, are you interested in reviewing? 


> slow query log analysis tool
> 
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Observability
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments: 
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch, 
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch, 
> csqldumpslow.py
>
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to 
> process the slow queries that are logged.  In the MySQL world, there's a tool 
> called mysqldumpslow, which processes a slow query log, abstracts the 
> parameters to prepared statements, and shows the queries which are causing 
> problems based on frequency.  The {{mysqldumpslow}} utillity shows an 
> aggregated count & time statistics spent on slow queries.  For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from 
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1  Time=4.32s (4s)  Lock=0.00s (0s)  Rows=0.0 (0), root[root]@localhost
>  insert into t2 select * from t1
> Count: 3  Time=2.53s (7s)  Lock=0.00s (0s)  Rows=0.0 (0), root[root]@localhost
>  insert into t2 select * from t1 limit N
> Count: 3  Time=2.13s (6s)  Lock=0.00s (0s)  Rows=0.0 (0), root[root]@localhost
>  insert into t1 select * from t1{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13935) Indexes creation should have IF EXISTS on its String representation

2017-10-20 Thread Jeff Jirsa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-13935:
--

Assignee: Javier Canillas

> Indexes creation should have IF EXISTS on its String representation
> ---
>
> Key: CASSANDRA-13935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13935
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL, Secondary Indexes
> Environment: Ubuntu 16.04.2 LTS
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
>Reporter: Javier Canillas
>Assignee: Javier Canillas
>Priority: Trivial
> Fix For: 3.11.x
>
> Attachments: 13935-3.0.txt, 13935-3.11.txt, 13935-trunk.txt
>
>
> I came across something that bothers me a lot. I'm using snapshots to backup 
> data from my Cassandra cluster in case something really bad happens (like 
> dropping a table or a keyspace).
> Exercising the recovery actions from those backups, I discover that the 
> schema put on the file "schema.cql" as a result of the snapshot has the 
> "CREATE IF NOT EXISTS" for the table, but not for the indexes.
> When restoring from snapshots, and relying on the execution of these schemas 
> to build up the table structure, everything seems fine for tables without 
> secondary indexes, but for the ones that make use of them, the execution of 
> these statements fail miserably.
> Here I paste a generated schema.cql content for a table with indexes:
> CREATE TABLE IF NOT EXISTS keyspace1.table1 (
>   id text PRIMARY KEY,
>   content text,
>   last_update_date date,
>   last_update_date_time timestamp)
>   WITH ID = f1045fc0-2f59-11e7-95ec-295c3c064920
>   AND bloom_filter_fp_chance = 0.01
>   AND dclocal_read_repair_chance = 0.1
>   AND crc_check_chance = 1.0
>   AND default_time_to_live = 864
>   AND gc_grace_seconds = 864000
>   AND min_index_interval = 128
>   AND max_index_interval = 2048
>   AND memtable_flush_period_in_ms = 0
>   AND read_repair_chance = 0.0
>   AND speculative_retry = '99PERCENTILE'
>   AND caching = { 'keys': 'NONE', 'rows_per_partition': 'NONE' }
>   AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 
> 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
>   AND compression = { 'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor' }
>   AND cdc = false
>   AND extensions = {  };
> CREATE INDEX table1_last_update_date_idx ON keyspace1.table1 
> (last_update_date);
> I think the last part should be:
> CREATE INDEX IF NOT EXISTS table1_last_update_date_idx ON keyspace1.table1 
> (last_update_date);



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13952) C* still sending WriteFailure instead of CDCWriteFailure

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213515#comment-16213515
 ] 

Jeff Jirsa commented on CASSANDRA-13952:


[~JoshuaMcKenzie] - am I reading you right that this is working-as-intended and 
not a bug?


> C* still sending WriteFailure instead of CDCWriteFailure
> 
>
> Key: CASSANDRA-13952
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13952
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
> Fix For: 4.0
>
>
> I've tested this setting setting cdc_total_space_in_mb: 1 and  with the 
> current python driver master
> {code}
> from cassandra.cluster import Cluster
> cluster = Cluster(protocol_version=5, allow_beta_protocol_version=True)
> session = cluster.connect()
> session.execute("""
> CREATE KEYSPACE IF NOT EXISTS %s
> WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '2' 
> }
> """ % "query_per_range")
> session.execute("""
> CREATE TABLE IF NOT EXISTS query_per_range.mytable_int (
> key_one int,
> key_two int,
> col1 text,
> col2 text,
> PRIMARY KEY ((key_one, key_two), col1)
> )  WITH cdc=true;
> """)
> for i in range(1000):
> session.execute("INSERT INTO query_per_range.mytable_int(key_one, 
> key_two, col1, col2) VALUES (%s, %s, %s, %s)",
> (i, i, str(i), str(i)))
> {code}
> I'm receiving a {{WriteFailure}} every time but I was expecting a 
> {{CDCWriteFailure}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13968) Cannot replace a live node on large clusters

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213511#comment-16213511
 ] 

Jeff Jirsa commented on CASSANDRA-13968:


Tagging [~jasobrown] and [~jkni] who have touched gossip quite a bit more than 
the rest of us - either of you feel like you're interested in reviewing?


> Cannot replace a live node on large clusters
> 
>
> Key: CASSANDRA-13968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13968
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.1.17, Ubuntu Trusty/Xenial (Linux 3.13, 4.4)
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>  Labels: gossip
> Attachments: 
> 0001-During-node-replacement-check-for-updates-in-the-tim.patch, 
> 0002-Only-fail-replacement-if-we-_know_-the-node-is-up.patch
>
>
> During forced node replacements we very frequently (~every time for large 
> clusters) see:
> {noformat}
> ERROR [main] 2017-10-17 06:54:35,680  CassandraDaemon.java:583 - Exception 
> encountered during startup
> java.lang.UnsupportedOperationException: Cannot replace a live node...
> {noformat}
> The old node is dead, the new node that is replacing it thinks it is dead (DN 
> state), and all other nodes think it is dead (all have the DN state). 
> However, I believe there are two bugs in the "is live" check that can cause 
> this error, namely that:
> 1. We sleep for 
> [BROADCAST_INTERVAL|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/service/StorageService.java#L905]
>  (hardcoded 60s on 2.1, on later version configurable but still 60s by 
> default), but 
> [check|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/service/StorageService.java#L919]
>  for an update in the last RING_DELAY seconds (typically set to 30s). When a 
> fresh node is joining, in my experience, [the 
> schema|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/service/StorageService.java#L859]
>  check almost immediately returns true after gossiping with seeds, so in 
> reality we do not even sleep for RING_DELAY. If operators increase ring delay 
> past broadcast_interval (as you might do if you think you are victim to the 
> second bug below), then you guarantee that you will always get the exception 
> because the gossip update is basically guaranteed to happen in the last 
> RING_DELAY seconds since you didn't sleep for that duration (you slept for 
> broadcast). For example if an operator sets ring delay to 300s, then the 
> check says "oh yea, the last update was 59 seconds ago, which is sooner than 
> 300s, so fail".
> 2. We don't actually check that the node is alive, we just check that a 
> gossip update has happened in the last X seconds. Sometimes with large 
> clusters nodes are still converging on the proper generation/version of a 
> dead node, and the "is live" check prevents an operator from replacing the 
> node until gossip has settled on the cluster regarding the dead node, which 
> for large clusters can take a really long time. This can be really hurtful to 
> availability in cloud environments and every time I've seen this error it's 
> the case that the new node believes that the old node is down (since 
> [markAlive|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/gms/Gossiper.java#L954]
>  [marks 
> dead|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/gms/Gossiper.java#L962]
>  first and then triggers a callback to 
> [realMarkAlive|https://github.com/apache/cassandra/blob/943db2488c8b62e1fbe03b132102f0e579c9ae17/src/java/org/apache/cassandra/gms/Gossiper.java#L975]
>  which never triggers because the old node is actually down).
> I think that #1 is definitely a bug, #2 might be considered an extra safety" 
> feature (that you don't allow replacement during gossip convergence), but 
> given that the operator took the effort to supply the replace_address flag, I 
> think it's prudent to only fail if we really know something is wrong.
> I've attached two patches against 2.1, one that fixes bug #1 and one that 
> fixes (imo) bug #2. I was thinking for #1 that we may want to prevent the 
> schema check from exiting the RING_DELAY sleep early but maybe it's just 
> better to backport configurable broadcast_interval and pick the maximum or 
> something. If we don't like the way I've worked around #2, maybe I could make 
> it an option that operators could turn on if they wanted? If folks are happy 
> with the approach I can attach patches for 2.2, 3.0, and 3.11.
> A relevant example of a log showing the

[jira] [Updated] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

2017-10-20 Thread Jeff Jirsa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13799:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

Thanks again [~malcolmt] . I've taken the liberty of committing to trunk as 
{{1c3a2d247a31eceef8893be7249119604a9c1b96}} with the test changes I mentioned 
in the previous comment. Really appreciate your help in raising this - looks 
like there are some other alerts on lgtm that we should address in the near 
future as well.



> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
>  Labels: static-analysis
> Fix For: 4.0
>
> Attachments: 13799-trunk.txt
>
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Fix some alerts raised by static analysis (lgtm.com)

2017-10-20 Thread jjirsa 
Repository: cassandra
Updated Branches:
  refs/heads/trunk 0b5088798 -> 1c3a2d247


Fix some alerts raised by static analysis (lgtm.com)

Patch by Malcolm Taylor; Reviewed by Jeff Jirsa for CASSANDRA-13799


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1c3a2d24
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1c3a2d24
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1c3a2d24

Branch: refs/heads/trunk
Commit: 1c3a2d247a31eceef8893be7249119604a9c1b96
Parents: 0b50887
Author: Malcolm Taylor 
Authored: Wed Aug 30 09:29:00 2017 +0100
Committer: Jeff Jirsa 
Committed: Fri Oct 20 16:44:22 2017 -0700

--
 CHANGES.txt |  1 +
 .../apache/cassandra/cql3/FieldIdentifier.java  |  2 +-
 .../org/apache/cassandra/cql3/RoleName.java |  2 +-
 .../cql3/conditions/ColumnCondition.java|  2 +-
 .../cql3/functions/FunctionResolver.java|  2 +-
 .../cassandra/index/sasi/plan/Expression.java   |  8 +++
 .../io/compress/CompressionMetadata.java| 14 +++---
 .../apache/cassandra/locator/TokenMetadata.java | 16 +++---
 .../cql3/functions/OperationFctsTest.java   | 10 ++--
 .../operations/InsertUpdateIfConditionTest.java | 32 ++--
 .../index/sasi/plan/ExpressionTest.java | 53 
 .../cassandra/stress/generate/Distribution.java |  8 +--
 .../cassandra/stress/generate/SeedManager.java  | 10 ++--
 .../stress/generate/values/Booleans.java|  2 +-
 14 files changed, 112 insertions(+), 50 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1c3a2d24/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6d84bc8..b2d6085 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,6 @@
 4.0
  * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728)
+ * Fix some alerts raised by static analysis (CASSANDRA-13799)
  * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593)
  * Add result set metadata to prepared statement MD5 hash calculation 
(CASSANDRA-10786)
  * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1c3a2d24/src/java/org/apache/cassandra/cql3/FieldIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/FieldIdentifier.java 
b/src/java/org/apache/cassandra/cql3/FieldIdentifier.java
index 9f72fc4..bdde98d 100644
--- a/src/java/org/apache/cassandra/cql3/FieldIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/FieldIdentifier.java
@@ -41,7 +41,7 @@ public class FieldIdentifier
  */
 public static FieldIdentifier forUnquoted(String text)
 {
-return new FieldIdentifier(convert(text.toLowerCase(Locale.US)));
+return new FieldIdentifier(convert(text == null ? null : 
text.toLowerCase(Locale.US)));
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1c3a2d24/src/java/org/apache/cassandra/cql3/RoleName.java
--
diff --git a/src/java/org/apache/cassandra/cql3/RoleName.java 
b/src/java/org/apache/cassandra/cql3/RoleName.java
index ce81fa9..b50c17d 100644
--- a/src/java/org/apache/cassandra/cql3/RoleName.java
+++ b/src/java/org/apache/cassandra/cql3/RoleName.java
@@ -25,7 +25,7 @@ public class RoleName
 
 public void setName(String name, boolean keepCase)
 {
-this.name = keepCase ? name : name.toLowerCase(Locale.US);
+this.name = keepCase ? name : (name == null ? name : 
name.toLowerCase(Locale.US));
 }
 
 public boolean hasName()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1c3a2d24/src/java/org/apache/cassandra/cql3/conditions/ColumnCondition.java
--
diff --git a/src/java/org/apache/cassandra/cql3/conditions/ColumnCondition.java 
b/src/java/org/apache/cassandra/cql3/conditions/ColumnCondition.java
index 24a8674..f3e0ee2 100644
--- a/src/java/org/apache/cassandra/cql3/conditions/ColumnCondition.java
+++ b/src/java/org/apache/cassandra/cql3/conditions/ColumnCondition.java
@@ -832,7 +832,7 @@ public abstract class ColumnCondition
 checkFalse(type.isCollection(), "Slice conditions are not 
supported on collections containing durations");
 checkFalse(type.isTuple(), "Slice conditions are not supported 
on tuples containing durations");
 checkFalse(type.isUDT(), "Slice conditions are not supported 
on UDTs containing durations");
-throw invalidRequest("Slice conditions are not supported on 
durations", operator);
+throw

[jira] [Commented] (CASSANDRA-13526) nodetool cleanup on KS with no replicas should remove old data, not silently complete

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213500#comment-16213500
 ] 

Jeff Jirsa commented on CASSANDRA-13526:


Hi [~jasonstack] Really appreciate your patience in the time it's taken me to 
back to this. I hope to review it this weekend.

[~krummas] / [~iamaleksey] - what do you folks think about versions here? 2.2 
or 3.0? 



> nodetool cleanup on KS with no replicas should remove old data, not silently 
> complete
> -
>
> Key: CASSANDRA-13526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13526
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jeff Jirsa
>Assignee: ZhaoYang
>  Labels: usability
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> From the user list:
> https://lists.apache.org/thread.html/5d49cc6bbc6fd2e5f8b12f2308a3e24212a55afbb441af5cb8cd4167@%3Cuser.cassandra.apache.org%3E
> If you have a multi-dc cluster, but some keyspaces not replicated to a given 
> DC, you'll be unable to run cleanup on those keyspaces in that DC, because 
> [the cleanup code will see no ranges and exit 
> early|https://github.com/apache/cassandra/blob/4cfaf85/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L427-L441]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

2017-10-20 Thread Ariel Weisberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213367#comment-16213367
 ] 

Ariel Weisberg commented on CASSANDRA-13123:


The new test that was added doesn't set up it's schema at all. It relies on the 
schema created during the previous test. Easy fix there.

The original test (before the new one was added) configures DatabaseDescriptor 
before loading the commit log (after which it can't be reloaded). The new test 
loads the commit log before this configuration occurs causing the original test 
to fail because the commit log configuration is wrong.

If you move the configuration into @BeforeClass the backpressure that is 
supposed to be in the original test doesn't occur and the test fails. No 
fragile test goes unpunished I guess. I think the might be an interaction with 
Byteman that I don't understand.

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.0.16, 3.11.2, 4.0
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13475) First version of pluggable storage engine API.

2017-10-20 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213287#comment-16213287
 ] 

Dikang Gu commented on CASSANDRA-13475:
---

[~spo...@gmail.com], back to your comment, here is a little write up of what I 
think about the responsibility and boundaries for a Cassandra storage engine.
=
A Cassandra pluggable storage engine is the component in Cassandra database 
sever that is responsible for managing how data is stored, both in memory and 
disk, and performing actual data I/O operations for a database, as well as 
enabling certain features sets that target a specific application need.

More concretely, the storage engine will be responsible for:

1. Physical Storage: This involves everything from supporting C* data types and 
table schema, as well as the format used for storing data to physical disk.
2. Query: storage engine will support point query and range query of data 
stored in the database.
3. Memory Caches: The storage engine may implement row cache or block cache, 
for query performance optimization.
4. Advanced Data Types: Like list/map/counter, it's up to storage engine 
whether to support the advanced data types or not.
5. Index Support: It's up to storage engine whether to support secondary index 
of the stored data or not.


The storage engine will NOT be responsible for any distributed or network 
features, like gossip, replication, streaming, repair, etc. Those features need 
to be implemented on top of the storage engine.
=

For CFS in the storage engine API, in the future, I'm thinking the CFS will 
simply be the hold of the metedata for a Cassandra table, so it's convenient to 
get necessary table metadata inside the storage engine, through the CFS class. 
But I'm open to change it, and load the metadata inside the storage engine 
instead.



> First version of pluggable storage engine API.
> --
>
> Key: CASSANDRA-13475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13475
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>
> In order to support pluggable storage engine, we need to define a unified 
> interface/API, which can allow us to plug in different storage engines for 
> different requirements. 
> In very high level, the storage engine interface should include APIs to:
> 1. Apply update into the engine.
> 2. Query data from the engine.
> 3. Stream data in/out to/from the engine.
> 4. Table operations, like create/drop/truncate a table, etc.
> 5. Various stats about the engine.
> I create this ticket to start the discussions about the interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

2017-10-20 Thread Jeff Jirsa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13799:
---
Labels: static-analysis  (was: )

> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
>  Labels: static-analysis
> Fix For: 4.x
>
> Attachments: 13799-trunk.txt
>
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213153#comment-16213153
 ] 

Jeff Jirsa commented on CASSANDRA-13799:


Thanks for the follow-up [~malcolmt] ! That looks like an appropriate fix.

There's also some unit tests (e.g. {{OperationFctsTest}} ) that rely on the 
old, incorrect logged strings. 

For that, I've made these changes, if you're ok with them:

{code}
diff --git 
a/test/unit/org/apache/cassandra/cql3/functions/OperationFctsTest.java 
b/test/unit/org/apache/cassandra/cql3/functions/OperationFctsTest.java
index 49c9e30..d27b746 100644
--- a/test/unit/org/apache/cassandra/cql3/functions/OperationFctsTest.java
+++ b/test/unit/org/apache/cassandra/cql3/functions/OperationFctsTest.java
@@ -292,17 +292,17 @@ public class OperationFctsTest extends CQLTester
row(2, (byte) 2, (short) 2, "test"));

 // tinyint, smallint and int could be used there so we need to 
disambiguate
-assertInvalidMessage("Ambiguous '+' operation: use type casts to 
disambiguate",
+assertInvalidMessage("Ambiguous '+' operation with args ? and 1: use 
type casts to disambiguate",
  "SELECT * FROM %s WHERE pk = ? + 1 AND c1 = 2", 
1);

-assertInvalidMessage("Ambiguous '+' operation: use type casts to 
disambiguate",
+assertInvalidMessage("Ambiguous '+' operation with args ? and 1: use 
type casts to disambiguate",
  "SELECT * FROM %s WHERE pk = 2 AND c1 = 2 AND c2 
= 1 * (? + 1)", 1);

 assertRows(execute("SELECT 1 + 1, v FROM %s WHERE pk = 2 AND c1 = 2"),
row(2, "test"));

 // As the output type is unknown the ? type cannot be determined
-assertInvalidMessage("Ambiguous '+' operation: use type casts to 
disambiguate",
+assertInvalidMessage("Ambiguous '+' operation with args 1 and ?: use 
type casts to disambiguate",
  "SELECT 1 + ?, v FROM %s WHERE pk = 2 AND c1 = 
2", 1);

 // As the prefered type for the constants is int, the returned type 
will be int
@@ -310,7 +310,7 @@ public class OperationFctsTest extends CQLTester
row(150, "test"));

 // As the output type is unknown the ? type cannot be determined
-assertInvalidMessage("Ambiguous '+' operation: use type casts to 
disambiguate",
+assertInvalidMessage("Ambiguous '+' operation with args ? and 50: use 
type casts to disambiguate",
  "SELECT ? + 50, v FROM %s WHERE pk = 2 AND c1 = 
2", 100);

 createTable("CREATE TABLE %s (a tinyint, b smallint, c int, d bigint, 
e float, f double, g varint, h decimal, PRIMARY KEY(a, b))"
@@ -672,7 +672,7 @@ public class OperationFctsTest extends CQLTester
 public void testWithNanAndInfinity() throws Throwable
 {
 createTable("CREATE TABLE %s (a int PRIMARY KEY, b double, c 
decimal)");
-assertInvalidMessage("Ambiguous '+' operation: use type casts to 
disambiguate",
+assertInvalidMessage("Ambiguous '+' operation with args ? and 1: use 
type casts to disambiguate",
  "INSERT INTO %S (a, b, c) VALUES (? + 1, ?, ?)", 
0, Double.NaN, BigDecimal.valueOf(1));

 execute("INSERT INTO %S (a, b, c) VALUES ((int) ? + 1, -?, ?)", 0, 
Double.NaN, BigDecimal.valueOf(1));{code}

And:

{code}
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/operations/InsertUpdateIfConditionTest.java
 
b/test/unit/org/apache/cassandra/cql3/validation/operations/InsertUpdateIfConditionTest.java
index da5c3bd..a0ae214 100644
--- 
a/test/unit/org/apache/cassandra/cql3/validation/operations/InsertUpdateIfConditionTest.java
+++ 
b/test/unit/org/apache/cassandra/cql3/validation/operations/InsertUpdateIfConditionTest.java
@@ -2038,13 +2038,13 @@ public class InsertUpdateIfConditionTest extends 
CQLTester
 {
 createTable(" CREATE TABLE %s (k int PRIMARY KEY, v int, d duration)");

-assertInvalidMessage("Slice conditions are not supported on durations",
+assertInvalidMessage("Slice conditions ( > ) are not supported on 
durations",
  "UPDATE %s SET v = 3 WHERE k = 0 IF d > 1s");
-assertInvalidMessage("Slice conditions are not supported on durations",
+assertInvalidMessage("Slice conditions ( >= ) are not supported on 
durations",
  "UPDATE %s SET v = 3 WHERE k = 0 IF d >= 1s");
-assertInvalidMessage("Slice conditions are not supported on durations",
+assertInvalidMessage("Slice conditions ( <= ) are not supported on 
durations",
  "UPDATE %s SET v = 3 WHERE k = 0 IF d <= 1s");
-assertInvalidMessage("Slice conditions are not supported on durations",
+assertInvalidMessage("Slice conditions ( < ) are not supported on 
durations",

[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault

2017-10-20 Thread Stefan Podkowinski (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213045#comment-16213045
 ] 

Stefan Podkowinski commented on CASSANDRA-13971:


The first step is to create a pluggable API that would abstract away how 
certificates and keystore instances are acquired. The default implementation 
would be compatible with the current way we configure and handle an existing 
keystore in the filesystem. The alternative Vault implementation will be 
implemented based on the Vault [REST 
API|https://www.vaultproject.io/api/index.html] (using Netty's HTTP support). 
We won't have to integrate any Vault code or library for that.

> Automatic certificate management using Vault
> 
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 4.x
>
>
> We've been adding security features during the last years to enable users to 
> secure their clusters, if they are willing to use them and do so correctly. 
> Some features are powerful and easy to work with, such as role based 
> authorization. Other features that require to manage a local keystore are 
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been 
> invented by us. We're just following Java standards there. But that doesn't 
> mean that we absolutely have to, if there are better options. I'd like to 
> give it a shoot and find out if we can automate certificate/key handling 
> (PKI) by using external APIs. In this case, the implementation will be based 
> on [Vault|https://vaultproject.io]. But certificate management services 
> offered by cloud providers may also be able to handle the use-case and I 
> intend to create a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault

2017-10-20 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212956#comment-16212956
 ] 

Jeff Jirsa commented on CASSANDRA-13971:


Really interesting idea. I need to read more to be sure, but it looks like 
you're proposing shipping an API to interact with vault, not actually extending 
or combining vault into the project, which would work around any issues with 
Vault's license (Mozilla license is in [category 
b|https://www.apache.org/legal/resolved.html#category-b], which isn't 
prohibited, but requires special handling). Is that correct reading of your 
proposal? 

> Automatic certificate management using Vault
> 
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 4.x
>
>
> We've been adding security features during the last years to enable users to 
> secure their clusters, if they are willing to use them and do so correctly. 
> Some features are powerful and easy to work with, such as role based 
> authorization. Other features that require to manage a local keystore are 
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been 
> invented by us. We're just following Java standards there. But that doesn't 
> mean that we absolutely have to, if there are better options. I'd like to 
> give it a shoot and find out if we can automate certificate/key handling 
> (PKI) by using external APIs. In this case, the implementation will be based 
> on [Vault|https://vaultproject.io]. But certificate management services 
> offered by cloud providers may also be able to handle the use-case and I 
> intend to create a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13475) First version of pluggable storage engine API.

2017-10-20 Thread Blake Eggleston (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212904#comment-16212904
 ] 

Blake Eggleston commented on CASSANDRA-13475:
-

This doesn’t abstract away the storage layer, so much as it bypasses it in a 
few places. Unfortunately, storage layer implementation details leak all over 
the project. So even though you’ve modified a few paths, C* will still be 
interacting directly with the C* storage layer in a lot of places.

> First version of pluggable storage engine API.
> --
>
> Key: CASSANDRA-13475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13475
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>
> In order to support pluggable storage engine, we need to define a unified 
> interface/API, which can allow us to plug in different storage engines for 
> different requirements. 
> In very high level, the storage engine interface should include APIs to:
> 1. Apply update into the engine.
> 2. Query data from the engine.
> 3. Stream data in/out to/from the engine.
> 4. Table operations, like create/drop/truncate a table, etc.
> 5. Various stats about the engine.
> I create this ticket to start the discussions about the interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"

2017-10-20 Thread Robert Stupp (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212809#comment-16212809
 ] 

Robert Stupp commented on CASSANDRA-13897:
--

+1 (assuming CI looks good)
One nit: make the new Config field a primitive instead of a boxed one.


> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
>Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is 
> not clear. This could probably be improved. Both of my two nodes return this 
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it, 
> then the error temporarily disappears. {{nodetool compress}} always returns 
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool 
> compact/rebuild/scrub, and nothing seems to remove the error. 
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
>   at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
>   at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"

2017-10-20 Thread Robert Stupp (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13897:
-
Status: Ready to Commit  (was: Patch Available)

> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
>Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is 
> not clear. This could probably be improved. Both of my two nodes return this 
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it, 
> then the error temporarily disappears. {{nodetool compress}} always returns 
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool 
> compact/rebuild/scrub, and nothing seems to remove the error. 
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
>   at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
>   at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
>   at 
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

2017-10-20 Thread Malcolm Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212791#comment-16212791
 ] 

Malcolm Taylor commented on CASSANDRA-13799:


I see that there was a test failure, sorry about that. The problem is in 
Distribution.java where the test was depending on a specific kind of floating 
point inaccuracy, and my change altered the number of iterations of the loop. 
This could be fixed by reverting the loop variable d from double back to float 
in the average() method. However, I think it would be cleaner to use an integer 
to control the loop, such as

public long average()
{
double sum = 0;
double d = 0d;
for (int count = 0; count < 51 ; count++)
{
sum += inverseCumProb(d);
d += 0.02d;
}
return (long) (sum / 51);
} 

> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
> Fix For: 4.x
>
> Attachments: 13799-trunk.txt
>
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13873) Ref bug in Scrub

2017-10-20 Thread Joel Knighton (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-13873:
--
Reproduced In: 3.11.0, 3.10, 4.0  (was: 3.10, 3.11.0, 4.0)
 Priority: Major  (was: Critical)

> Ref bug in Scrub
> 
>
> Key: CASSANDRA-13873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13873
> Project: Cassandra
>  Issue Type: Bug
>Reporter: T Jake Luciani
>Assignee: Joel Knighton
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> I'm hitting a Ref bug when many scrubs run against a node.  This doesn't 
> happen on 3.0.X.  I'm not sure if/if not this happens with compactions too 
> but I suspect it does.
> I'm not seeing any Ref leaks or double frees.
> To Reproduce:
> {quote}
> ./tools/bin/cassandra-stress write n=10m -rate threads=100
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> {quote}
> Eventually in the logs you get:
> WARN  [RMI TCP Connection(4)-127.0.0.1] 2017-09-14 15:51:26,722 
> NoSpamLogger.java:97 - Spinning trying to capture readers 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-32-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-31-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-29-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-27-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-26-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-20-big-Data.db')],
> *released: 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db')],*
>  
> This released table has a selfRef of 0 but is in the Tracker



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13873) Ref bug in Scrub

2017-10-20 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212774#comment-16212774
 ] 

Joel Knighton commented on CASSANDRA-13873:
---

+1 - I'd like to introduce as few changes as possible to older versions here. 
That combination sounds good to me. Do you want to prepare the patch for older 
versions or would you like me to?

> Ref bug in Scrub
> 
>
> Key: CASSANDRA-13873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13873
> Project: Cassandra
>  Issue Type: Bug
>Reporter: T Jake Luciani
>Assignee: Joel Knighton
>Priority: Critical
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> I'm hitting a Ref bug when many scrubs run against a node.  This doesn't 
> happen on 3.0.X.  I'm not sure if/if not this happens with compactions too 
> but I suspect it does.
> I'm not seeing any Ref leaks or double frees.
> To Reproduce:
> {quote}
> ./tools/bin/cassandra-stress write n=10m -rate threads=100
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> {quote}
> Eventually in the logs you get:
> WARN  [RMI TCP Connection(4)-127.0.0.1] 2017-09-14 15:51:26,722 
> NoSpamLogger.java:97 - Spinning trying to capture readers 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-32-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-31-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-29-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-27-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-26-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-20-big-Data.db')],
> *released: 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db')],*
>  
> This released table has a selfRef of 0 but is in the Tracker



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13873) Ref bug in Scrub

2017-10-20 Thread Marcus Eriksson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212759#comment-16212759
 ] 

Marcus Eriksson commented on CASSANDRA-13873:
-

The problem seems to be that we don't grab a reference before starting the 
compaction everywhere - we do it in 
[CompactionTask|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/CompactionTask.java#L177],
 but not during 
[cleanup|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1095-L1098]
 for example.

[~jkni] I think we should do your fix (and also remove the external 
ref-grabbing in CompactionTask), but maybe only in trunk? It feels a tiny bit 
safer to not touch {{LifecycleTransaction}}/{{CompactionTask}} in older 
versions and instead take the ref outside where we need it, but I'm not 
entirely convinced, wdyt?


> Ref bug in Scrub
> 
>
> Key: CASSANDRA-13873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13873
> Project: Cassandra
>  Issue Type: Bug
>Reporter: T Jake Luciani
>Assignee: Joel Knighton
>Priority: Critical
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> I'm hitting a Ref bug when many scrubs run against a node.  This doesn't 
> happen on 3.0.X.  I'm not sure if/if not this happens with compactions too 
> but I suspect it does.
> I'm not seeing any Ref leaks or double frees.
> To Reproduce:
> {quote}
> ./tools/bin/cassandra-stress write n=10m -rate threads=100
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> {quote}
> Eventually in the logs you get:
> WARN  [RMI TCP Connection(4)-127.0.0.1] 2017-09-14 15:51:26,722 
> NoSpamLogger.java:97 - Spinning trying to capture readers 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-32-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-31-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-29-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-27-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-26-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-20-big-Data.db')],
> *released: 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db')],*
>  
> This released table has a selfRef of 0 but is in the Tracker



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13873) Ref bug in Scrub

2017-10-20 Thread Marcus Eriksson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13873:

Reproduced In: 3.11.0, 3.10, 4.0  (was: 3.10, 3.11.0, 4.0)
Fix Version/s: (was: 2.1.x)

correction, there is only a tiny race just as we finish up compactions in 2.1 - 
not worth fixing at this point

> Ref bug in Scrub
> 
>
> Key: CASSANDRA-13873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13873
> Project: Cassandra
>  Issue Type: Bug
>Reporter: T Jake Luciani
>Assignee: Joel Knighton
>Priority: Critical
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> I'm hitting a Ref bug when many scrubs run against a node.  This doesn't 
> happen on 3.0.X.  I'm not sure if/if not this happens with compactions too 
> but I suspect it does.
> I'm not seeing any Ref leaks or double frees.
> To Reproduce:
> {quote}
> ./tools/bin/cassandra-stress write n=10m -rate threads=100
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> #Ctrl-C
> ./bin/nodetool scrub
> {quote}
> Eventually in the logs you get:
> WARN  [RMI TCP Connection(4)-127.0.0.1] 2017-09-14 15:51:26,722 
> NoSpamLogger.java:97 - Spinning trying to capture readers 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-32-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-31-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-29-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-27-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-26-big-Data.db'),
>  
> BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-20-big-Data.db')],
> *released: 
> [BigTableReader(path='/home/jake/workspace/cassandra2/data/data/keyspace1/standard1-2eb5c780998311e79e09311efffdcd17/mc-5-big-Data.db')],*
>  
> This released table has a selfRef of 0 but is in the Tracker



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13971) Automatic certificate management using Vault

2017-10-20 Thread Stefan Podkowinski (JIRA) 
Stefan Podkowinski created CASSANDRA-13971:
--

 Summary: Automatic certificate management using Vault
 Key: CASSANDRA-13971
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
 Project: Cassandra
  Issue Type: Improvement
  Components: Streaming and Messaging
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
 Fix For: 4.x


We've been adding security features during the last years to enable users to 
secure their clusters, if they are willing to use them and do so correctly. 
Some features are powerful and easy to work with, such as role based 
authorization. Other features that require to manage a local keystore are 
rather painful to deal with. Think about setting up SSL..

To be fair, keystore related issues and certificate handling hasn't been 
invented by us. We're just following Java standards there. But that doesn't 
mean that we absolutely have to, if there are better options. I'd like to give 
it a shoot and find out if we can automate certificate/key handling (PKI) by 
using external APIs. In this case, the implementation will be based on 
[Vault|https://vaultproject.io]. But certificate management services offered by 
cloud providers may also be able to handle the use-case and I intend to create 
a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

2017-10-20 Thread Malcolm Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212313#comment-16212313
 ] 

Malcolm Taylor commented on CASSANDRA-13799:


Thanks for the review [~jjirsa], and thanks for the improvement to 
testBoundHashCode(). Your changes look fine to me.

> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
> Fix For: 4.x
>
> Attachments: 13799-trunk.txt
>
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org