[jira] [Comment Edited] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[zhaoyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119329#comment-16119329
 ] 

zhaoyan edited comment on CASSANDRA-13363 at 8/9/17 5:56 AM:
-

I apply one patch here:

https://github.com/apache/cassandra/pull/137


was (Author: zhaoyan):
I apply one patch here:

https://github.com/johnyannj/cassandra/pull/1

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119434#comment-16119434
 ] 

ASF GitHub Bot commented on CASSANDRA-13741:


GitHub user ghatwala opened a pull request:

https://github.com/apache/cassandra/pull/138

Upgrading lz4 version to 1.4.0 in trunk

Refer JIRA ticket link : 
https://issues.apache.org/jira/browse/CASSANDRA-13741 for discussions .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ghatwala/cassandra Upgrading_lz4_version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #138


commit 7df4b819264e141a030d97ebb6de6018cd66d7c4
Author: ghatwala 
Date:   2017-08-09T05:19:49Z

Upgrading lz4 version to 1.4.0 in trunk
Refer JIRA ticket link : 
https://issues.apache.org/jira/browse/CASSANDRA-13741




> Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
> -
>
> Key: CASSANDRA-13741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13741
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Libraries
>Reporter: Amitkumar Ghatwal
>Assignee: Michael Kjellman
> Fix For: 4.x
>
>
> Hi All,
> The latest lz4-java library has been released 
> (https://github.com/lz4/lz4-java/releases) and uploaded to maven central . 
> Please replace in mainline the current version ( 1.3.0) with the latest one ( 
> 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/
> Adding : [~ReiOdaira].
> Regards,
> Amit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar

[Amitkumar Ghatwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119435#comment-16119435
 ] 

Amitkumar Ghatwal commented on CASSANDRA-13741:
---

Thanks- [~mkjellman] for taking this up . PR link for your referral please - 
https://github.com/apache/cassandra/pull/138 

> Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
> -
>
> Key: CASSANDRA-13741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13741
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Libraries
>Reporter: Amitkumar Ghatwal
>Assignee: Michael Kjellman
> Fix For: 4.x
>
>
> Hi All,
> The latest lz4-java library has been released 
> (https://github.com/lz4/lz4-java/releases) and uploaded to maven central . 
> Please replace in mainline the current version ( 1.3.0) with the latest one ( 
> 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/
> Adding : [~ReiOdaira].
> Regards,
> Amit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13363:
---
Status: Patch Available  (was: Open)

Marking patch available


> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13363:
---
Fix Version/s: 4.x
   3.11.x
   3.0.x

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-13363:
--

Assignee: zhaoyan

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Assignee: zhaoyan
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-12435) Cassandra timeout when using secondary index in cluster mode.

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa resolved CASSANDRA-12435.

Resolution: Duplicate

Apparently a dupe of CASSANDRA-13363 , which has a patch-available (though not 
reviewed/tested).

> Cassandra timeout when using secondary index in cluster mode.
> -
>
> Key: CASSANDRA-12435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12435
> Project: Cassandra
>  Issue Type: Bug
>  Components: Secondary Indexes
> Environment: C* 3.5 | java-driver|AWS g2.2xlarge|ubuntu 14.04
>Reporter: Shijian Liu
> Fix For: 3.11.x
>
>
> I have a 3-node cluster with a keyspace with RF=3, then I created a table 
> like below and inserted some data(about 10MB).
> {code}
> CREATE TABLE vehicle_fuzzy_plate (
> day int,
> hour int,
> repo int,
> sensor_id int,
> ts timestamp,
> id timeuuid,
> plate_text_pattern_70 text,
> plate_text_pattern_10 text,
> plate_text_pattern_11 text,
> plate_text_pattern_12 text,
> plate_text_pattern_13 text,
> plate_text_pattern_14 text,
> plate_text_pattern_15 text,
> plate_text_pattern_16 text,
> plate_text_pattern_20 text,
> plate_text_pattern_21 text,
> plate_text_pattern_22 text,
> plate_text_pattern_23 text,
> plate_text_pattern_24 text,
> plate_text_pattern_25 text,
> plate_text_pattern_30 text,
> plate_text_pattern_31 text,
> plate_text_pattern_32 text,
> plate_text_pattern_33 text,
> plate_text_pattern_34 text,
> plate_text_pattern_40 text,
> plate_text_pattern_41 text,
> plate_text_pattern_42 text,
> plate_text_pattern_43 text,
> plate_text_pattern_50 text,
> plate_text_pattern_51 text,
> plate_text_pattern_52 text,
> plate_text_pattern_60 text,
> plate_text_pattern_61 text,
> PRIMARY KEY (day, repo, sensor_id, ts, id)
> ) WITH CLUSTERING ORDER BY (repo ASC, sensor_id ASC, ts ASC, id ASC);
> CREATE INDEX plate_text_pattern_10_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_10);
> CREATE INDEX plate_text_pattern_11_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_11);
> CREATE INDEX plate_text_pattern_12_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_12);
> CREATE INDEX plate_text_pattern_13_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_13);
> CREATE INDEX plate_text_pattern_14_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_14);
> CREATE INDEX plate_text_pattern_15_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_15);
> CREATE INDEX plate_text_pattern_16_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_16);
> CREATE INDEX plate_text_pattern_20_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_20);
> CREATE INDEX plate_text_pattern_21_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_21);
> CREATE INDEX plate_text_pattern_22_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_22);
> CREATE INDEX plate_text_pattern_23_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_23);
> CREATE INDEX plate_text_pattern_24_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_24);
> CREATE INDEX plate_text_pattern_25_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_25);
> CREATE INDEX plate_text_pattern_30_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_30);
> CREATE INDEX plate_text_pattern_31_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_31);
> CREATE INDEX plate_text_pattern_32_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_32);
> CREATE INDEX plate_text_pattern_33_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_33);
> CREATE INDEX plate_text_pattern_34_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_34);
> CREATE INDEX plate_text_pattern_40_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_40);
> CREATE INDEX plate_text_pattern_41_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_41);
> CREATE INDEX plate_text_pattern_42_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_42);
> CREATE INDEX plate_text_pattern_43_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_43);
> CREATE INDEX plate_text_pattern_50_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_50);
> CREATE INDEX plate_text_pattern_51_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_51);
> CREATE INDEX plate_text_pattern_52_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_52);
> CREATE INDEX plate_text_pattern_60_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_60);
> CREATE INDEX plate_text_pattern_61_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_61);
> CREATE INDEX plate_text_pattern_70_idx ON vehicle_fuzzy_plate 
> (plate_text_pattern_70);
> {code}
> After that, I send some queries through Java driver with Consistency=Quorum 
> one by one, for example,
> {code}
> SELECT id,sensor_id,ts FROM vehicle_fuzzy_plate WHERE day IN 
> 

[jira] [Commented] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119377#comment-16119377
 ] 

ASF GitHub Bot commented on CASSANDRA-13363:


GitHub user johnyannj opened a pull request:

https://github.com/apache/cassandra/pull/137

CASSANDRA-13363/CASSANDRA-12435 fix index flag before not be consistent 
with follow data content

fix CASSANDRA-13363/CASSANDRA-12435

when the command will be executed by local node and target node.
the command.index will be reload by local thread.

when code reach indexFlag(command.index.isPresent())  is false
but reach if (command.index.isPresent()) , it is changed to true, because 
the "command.index" has been loaded by local execute thread.

the command.index can be reload by target node, so it's serialize is 
optional。 but the flag must be consistent with follow data content.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnyannj/cassandra johnyannj-patch-fix-13363

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #137


commit 5050ff51b1f62fe7a678be01f79ff962a0fd6caa
Author: zhaoyan 
Date:   2017-08-09T02:35:46Z

Update ReadCommand.java

fix CASSANDRA-13363/CASSANDRA-12435

when the command will be executed by local node and target node.
the command.index will be reload by local thread.

when code reach indexFlag(command.index.isPresent())  is false
but reach if (command.index.isPresent()) , it is changed to true, because 
the "command.index" has been loaded by local execute thread.

the command.index can be reload by target node, so it's serialize is 
optional。 but the flag must be consistent with follow data content.




> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Priority: Critical
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[zhaoyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119329#comment-16119329
 ] 

zhaoyan edited comment on CASSANDRA-13363 at 8/9/17 3:39 AM:
-

I apply one patch here:

https://github.com/johnyannj/cassandra/pull/1


was (Author: zhaoyan):
I give one patch here:

https://github.com/johnyannj/cassandra/pull/1

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Priority: Critical
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[zhaoyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119329#comment-16119329
 ] 

zhaoyan edited comment on CASSANDRA-13363 at 8/9/17 3:38 AM:
-

I give one patch here:

https://github.com/johnyannj/cassandra/pull/1


was (Author: zhaoyan):
I apply one patch:

https://github.com/johnyannj/cassandra/pull/1

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Priority: Critical
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13363) java.lang.ArrayIndexOutOfBoundsException: null

[zhaoyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119329#comment-16119329
 ] 

zhaoyan commented on CASSANDRA-13363:
-

I apply one patch:

https://github.com/johnyannj/cassandra/pull/1

> java.lang.ArrayIndexOutOfBoundsException: null
> --
>
> Key: CASSANDRA-13363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13363
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS 6, Cassandra 3.10
>Reporter: Artem Rokhin
>Priority: Critical
>
> Constantly see this error in the log without any additional information or a 
> stack trace.
> {code}
> Exception in thread Thread[MessagingService-Incoming-/10.0.1.26,5,main]
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: null
> {code}
> Logger: org.apache.cassandra.service.CassandraDaemon
> Thrdead: MessagingService-Incoming-/10.0.1.12
> Method: uncaughtException
> File: CassandraDaemon.java
> Line: 229



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
--
Attachment: (was: 13740-3.0.15.txt)

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
--
Attachment: 13740-3.0.15.txt

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
--
Attachment: (was: 13740-2_3.0.15.txt)

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119106#comment-16119106
 ] 

Jaydeepkumar Chovatia edited comment on CASSANDRA-13740 at 8/8/17 10:19 PM:


Please find pull request against Cassandra-3.0 here: 
https://github.com/apache/cassandra/pull/136/commits/993d5891e69f5cda402ae2158e06914f653a644d


was (Author: chovatia.jayd...@gmail.com):
Please find pull request here: 
https://github.com/apache/cassandra/pull/136/commits/993d5891e69f5cda402ae2158e06914f653a644d

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119106#comment-16119106
 ] 

Jaydeepkumar Chovatia commented on CASSANDRA-13740:
---

Please find pull request here: 
https://github.com/apache/cassandra/pull/136/commits/993d5891e69f5cda402ae2158e06914f653a644d

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-2_3.0.15.txt, 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119094#comment-16119094
 ] 

ASF GitHub Bot commented on CASSANDRA-13740:


GitHub user jaydeepkumar1984 opened a pull request:

https://github.com/apache/cassandra/pull/136

CASSANDRA-13740 orphan hint file get created



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaydeepkumar1984/cassandra 13740-3.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #136


commit 993d5891e69f5cda402ae2158e06914f653a644d
Author: Jaydeepkumar Chovatia 
Date:   2017-08-03T22:34:26Z

CASSANDRA-13740 orphan hint file get created




> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13740-2_3.0.15.txt, 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13750) Counter digests include local data

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13750:

Status: Patch Available  (was: Open)

|[trunk|https://github.com/bdeggleston/cassandra/tree/13750-trunk]| 
[utest|https://circleci.com/gh/bdeggleston/cassandra/81] / [dtest 
(pending)|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/172/]|
|[3.11|https://github.com/bdeggleston/cassandra/tree/13750-3.11] | 
[utest|https://circleci.com/gh/bdeggleston/cassandra/80]|
|[3.0|https://github.com/bdeggleston/cassandra/tree/13750-3.0] | 
[utest|https://circleci.com/gh/bdeggleston/cassandra/79] |

> Counter digests include local data
> --
>
> Key: CASSANDRA-13750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13750
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> In 3.x+, the raw counter value bytes are used when hashing counters for reads 
> and repair, including local shard data, which is removed when streamed. This 
> leads to constant digest mismatches and repair overstreaming.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118945#comment-16118945
 ] 

Michael Kjellman commented on CASSANDRA-13741:
--

Yes, I'll take this. [~amitkumar_ghatwal] do you have a trunk patch for this 
somewhere already that I'm missing?

> Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
> -
>
> Key: CASSANDRA-13741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13741
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Libraries
>Reporter: Amitkumar Ghatwal
>Assignee: Michael Kjellman
> Fix For: 4.x
>
>
> Hi All,
> The latest lz4-java library has been released 
> (https://github.com/lz4/lz4-java/releases) and uploaded to maven central . 
> Please replace in mainline the current version ( 1.3.0) with the latest one ( 
> 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/
> Adding : [~ReiOdaira].
> Regards,
> Amit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar

[Michael Kjellman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Kjellman reassigned CASSANDRA-13741:


Assignee: Michael Kjellman

> Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
> -
>
> Key: CASSANDRA-13741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13741
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Libraries
>Reporter: Amitkumar Ghatwal
>Assignee: Michael Kjellman
> Fix For: 4.x
>
>
> Hi All,
> The latest lz4-java library has been released 
> (https://github.com/lz4/lz4-java/releases) and uploaded to maven central . 
> Please replace in mainline the current version ( 1.3.0) with the latest one ( 
> 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/
> Adding : [~ReiOdaira].
> Regards,
> Amit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11748) Schema version mismatch may leads to Casandra OOM at bootstrap during a rolling upgrade process

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11748:
--
Reviewer: Aleksey Yeschenko

> Schema version mismatch may leads to Casandra OOM at bootstrap during a 
> rolling upgrade process
> ---
>
> Key: CASSANDRA-11748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11748
> Project: Cassandra
>  Issue Type: Bug
> Environment: Rolling upgrade process from 1.2.19 to 2.0.17. 
> CentOS 6.6
> Occurred in different C* node of different scale of deployment (2G ~ 5G)
>Reporter: Michael Fong
>Assignee: Matt Byrd
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> We have observed multiple times when a multi-node C* (v2.0.17) cluster ran 
> into OOM in bootstrap during a rolling upgrade process from 1.2.19 to 2.0.17. 
> Here is the simple guideline of our rolling upgrade process
> 1. Update schema on a node, and wait until all nodes to be in schema version 
> agreemnt - via nodetool describeclulster
> 2. Restart a Cassandra node
> 3. After restart, there is a chance that the the restarted node has different 
> schema version.
> 4. All nodes in cluster start to rapidly exchange schema information, and any 
> of node could run into OOM. 
> The following is the system.log that occur in one of our 2-node cluster test 
> bed
> --
> Before rebooting node 2:
> Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326 
> MigrationManager.java (line 328) Gossiping my schema version 
> 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> Node 2: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,122 
> MigrationManager.java (line 328) Gossiping my schema version 
> 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> After rebooting node 2, 
> Node 2: DEBUG [main] 2016-04-19 11:18:18,016 MigrationManager.java (line 328) 
> Gossiping my schema version f5270873-ba1f-39c7-ab2e-a86db868b09b
> The node2  keeps submitting the migration task over 100+ times to the other 
> node.
> INFO [GossipStage:1] 2016-04-19 11:18:18,261 Gossiper.java (line 1011) Node 
> /192.168.88.33 has restarted, now UP
> INFO [GossipStage:1] 2016-04-19 11:18:18,262 TokenMetadata.java (line 414) 
> Updating topology for /192.168.88.33
> ...
> DEBUG [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line 
> 102) Submitting migration task for /192.168.88.33
> ... ( over 100+ times)
> --
> On the otherhand, Node 1 keeps updating its gossip information, followed by 
> receiving and submitting migrationTask afterwards: 
> INFO [RequestResponseStage:3] 2016-04-19 11:18:18,333 Gossiper.java (line 
> 978) InetAddress /192.168.88.34 is now UP
> ...
> DEBUG [MigrationStage:1] 2016-04-19 11:18:18,496 
> MigrationRequestVerbHandler.java (line 41) Received migration request from 
> /192.168.88.34.
> …… ( over 100+ times)
> DEBUG [OptionalTasks:1] 2016-04-19 11:19:18,337 MigrationManager.java (line 
> 127) submitting migration task for /192.168.88.34
> .  (over 50+ times)
> On the side note, we have over 200+ column families defined in Cassandra 
> database, which may related to this amount of rpc traffic.
> P.S.2 The over requested schema migration task will eventually have 
> InternalResponseStage performing schema merge operation. Since this operation 
> requires a compaction for each merge and is much slower to consume. Thus, the 
> back-pressure of incoming schema migration content objects consumes all of 
> the heap space and ultimately ends up OOM!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11748) Schema version mismatch may leads to Casandra OOM at bootstrap during a rolling upgrade process

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11748:
--
Status: Patch Available  (was: Open)

> Schema version mismatch may leads to Casandra OOM at bootstrap during a 
> rolling upgrade process
> ---
>
> Key: CASSANDRA-11748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11748
> Project: Cassandra
>  Issue Type: Bug
> Environment: Rolling upgrade process from 1.2.19 to 2.0.17. 
> CentOS 6.6
> Occurred in different C* node of different scale of deployment (2G ~ 5G)
>Reporter: Michael Fong
>Assignee: Matt Byrd
>Priority: Critical
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> We have observed multiple times when a multi-node C* (v2.0.17) cluster ran 
> into OOM in bootstrap during a rolling upgrade process from 1.2.19 to 2.0.17. 
> Here is the simple guideline of our rolling upgrade process
> 1. Update schema on a node, and wait until all nodes to be in schema version 
> agreemnt - via nodetool describeclulster
> 2. Restart a Cassandra node
> 3. After restart, there is a chance that the the restarted node has different 
> schema version.
> 4. All nodes in cluster start to rapidly exchange schema information, and any 
> of node could run into OOM. 
> The following is the system.log that occur in one of our 2-node cluster test 
> bed
> --
> Before rebooting node 2:
> Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326 
> MigrationManager.java (line 328) Gossiping my schema version 
> 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> Node 2: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,122 
> MigrationManager.java (line 328) Gossiping my schema version 
> 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> After rebooting node 2, 
> Node 2: DEBUG [main] 2016-04-19 11:18:18,016 MigrationManager.java (line 328) 
> Gossiping my schema version f5270873-ba1f-39c7-ab2e-a86db868b09b
> The node2  keeps submitting the migration task over 100+ times to the other 
> node.
> INFO [GossipStage:1] 2016-04-19 11:18:18,261 Gossiper.java (line 1011) Node 
> /192.168.88.33 has restarted, now UP
> INFO [GossipStage:1] 2016-04-19 11:18:18,262 TokenMetadata.java (line 414) 
> Updating topology for /192.168.88.33
> ...
> DEBUG [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line 
> 102) Submitting migration task for /192.168.88.33
> ... ( over 100+ times)
> --
> On the otherhand, Node 1 keeps updating its gossip information, followed by 
> receiving and submitting migrationTask afterwards: 
> INFO [RequestResponseStage:3] 2016-04-19 11:18:18,333 Gossiper.java (line 
> 978) InetAddress /192.168.88.34 is now UP
> ...
> DEBUG [MigrationStage:1] 2016-04-19 11:18:18,496 
> MigrationRequestVerbHandler.java (line 41) Received migration request from 
> /192.168.88.34.
> …… ( over 100+ times)
> DEBUG [OptionalTasks:1] 2016-04-19 11:19:18,337 MigrationManager.java (line 
> 127) submitting migration task for /192.168.88.34
> .  (over 50+ times)
> On the side note, we have over 200+ column families defined in Cassandra 
> database, which may related to this amount of rpc traffic.
> P.S.2 The over requested schema migration task will eventually have 
> InternalResponseStage performing schema merge operation. Since this operation 
> requires a compaction for each merge and is much slower to consume. Thus, the 
> back-pressure of incoming schema migration content objects consumes all of 
> the heap space and ultimately ends up OOM!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13750) Counter digests include local data

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13750:
--
Reviewer: Aleksey Yeschenko

> Counter digests include local data
> --
>
> Key: CASSANDRA-13750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13750
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> In 3.x+, the raw counter value bytes are used when hashing counters for reads 
> and repair, including local shard data, which is removed when streamed. This 
> leads to constant digest mismatches and repair overstreaming.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13751) Race / ref leak in PendingRepairManager

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13751:

Reviewer: Marcus Eriksson
  Status: Patch Available  (was: Open)

[trunk|https://github.com/bdeggleston/cassandra/tree/13751]
[utest|https://circleci.com/gh/bdeggleston/cassandra/76]
[dtest 
(pending)|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/171/]

> Race / ref leak in PendingRepairManager
> ---
>
> Key: CASSANDRA-13751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13751
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> PendingRepairManager#getScanners has an assertion that confirms an sstable 
> is, in fact, marked as pending repair. Since validation compactions don't use 
> the same concurrency controls as proper compactions, they can race with 
> promotion/demotion compactions and end up getting assertion errors when the 
> pending repair id is changed while the scanners are being acquired. Also, 
> error handling in PendingRepairManager and CompactionStrategyManager leaks 
> refs when this happens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13751) Race / ref leak in PendingRepairManager

[Blake Eggleston (JIRA)]

Blake Eggleston created CASSANDRA-13751:
---

 Summary: Race / ref leak in PendingRepairManager
 Key: CASSANDRA-13751
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13751
 Project: Cassandra
  Issue Type: Bug
Reporter: Blake Eggleston
Assignee: Blake Eggleston
Priority: Minor
 Fix For: 4.0


PendingRepairManager#getScanners has an assertion that confirms an sstable is, 
in fact, marked as pending repair. Since validation compactions don't use the 
same concurrency controls as proper compactions, they can race with 
promotion/demotion compactions and end up getting assertion errors when the 
pending repair id is changed while the scanners are being acquired. Also, error 
handling in PendingRepairManager and CompactionStrategyManager leaks refs when 
this happens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13750) Counter digests include local data

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13750:

Description: In 3.x+, the raw counter value bytes are used when hashing 
counters for reads and repair, including local shard data, which is removed 
when streamed. This leads to constant digest mismatches and repair 
overstreaming.  (was: In 3.x+, the raw counter value bytes are used when 
hashing counters for reads and repair, including local shard data, which is 
removed when streamed. This leads to constant digest mismatches and repair 
overstreaming. This patch switches to using CounterContext#updateDigest for 
counter cells, which fixes this problem by not hashing the header.

This will cause a temporary increase in digest mismatches during minor 
upgrades.)

> Counter digests include local data
> --
>
> Key: CASSANDRA-13750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13750
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> In 3.x+, the raw counter value bytes are used when hashing counters for reads 
> and repair, including local shard data, which is removed when streamed. This 
> leads to constant digest mismatches and repair overstreaming.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13750) Counter digests include local data

[Blake Eggleston (JIRA)]

Blake Eggleston created CASSANDRA-13750:
---

 Summary: Counter digests include local data
 Key: CASSANDRA-13750
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13750
 Project: Cassandra
  Issue Type: Bug
Reporter: Blake Eggleston
Assignee: Blake Eggleston
Priority: Minor
 Fix For: 4.0, 3.0.x, 3.11.x


In 3.x+, the raw counter value bytes are used when hashing counters for reads 
and repair, including local shard data, which is removed when streamed. This 
leads to constant digest mismatches and repair overstreaming. This patch 
switches to using CounterContext#updateDigest for counter cells, which fixes 
this problem by not hashing the header.

This will cause a temporary increase in digest mismatches during minor upgrades.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118667#comment-16118667
 ] 

Aleksey Yeschenko commented on CASSANDRA-13043:
---

I'll happily review a patch if you fix counter leader election, but will 
unlikely fix it myself soon, sorry.

> UnavailabeException caused by counter writes forwarded to leaders without 
> complete cluster view
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> 

[jira] [Commented] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118638#comment-16118638
 ] 

Aleksey Yeschenko commented on CASSANDRA-13043:
---

There also seems to be a potential edge-case bug with local consistency levels 
and leader picking, but overall this is some very old code that probably needs 
another look ({{StoraProxy}} counters related methods). 

> UnavailabeException caused by counter writes forwarded to leaders without 
> complete cluster view
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 

[jira] [Updated] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13043:
--
Summary: UnavailabeException caused by counter writes forwarded to leaders 
without complete cluster view  (was: Unable to achieve CL while applying 
counters from commitlog)

> UnavailabeException caused by counter writes forwarded to leaders without 
> complete cluster view
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> 

[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118625#comment-16118625
 ] 

Aleksey Yeschenko commented on CASSANDRA-13043:
---

bq. I don't understand why the replica node (the node where this exception is 
triggered) is checking for Consistency Level though, but I guess it might be a 
corner case of how counters are handled.

The coordinator for the counter write - the initial point of contact - will 
pick the leader first, the node will end up owning the write. The leader would 
perform the blocking read before write, then propagate the applied change to 
the rest of the replica set.

Even if the coordinator is a replica itself, it might still elect another one 
as a leader - to balance the load.

Picking the leader is pretty simplistic. We pick a random node in the local DC, 
if any. If none, we just go and pick the closest replica to the coordinator, as 
measured by the snitch.

As for checking for consistency level, we do it twice. Once on the coordinator, 
to allow a faster response if we know in advance that we cannot fulfil the CL, 
and then again on the leader itself, as part of {{StorageProxy.performWrite()}}.

So, yes,
1) Availability check happens on the leader, but also on the coordinator, 
before that.
2) The leader is getting the requests forwarded from the coordinator, not the 
client, explaining why despite the node not yet listening for native protocol 
clients can still trigger the UE.



> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> 

[jira] [Comment Edited] (CASSANDRA-13594) Use an ExecutorService for repair commands instead of new Thread(..).start()

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118439#comment-16118439
 ] 

Ariel Weisberg edited comment on CASSANDRA-13594 at 8/8/17 4:04 PM:


I've noticed that short_read test seems to time out quite a bit. I'll run it 
locally a bit and see if it reproduces. I think I did before with no luck. This 
issue might only appear in Apache Jenkins.


was (Author: aweisberg):
I've noticed that short_read test seems to time out quite a bit. I'll run it 
locally a bit and see if it reproduces. I think I did before with no luck. This 
an issue might only appear in Apache Jenkins.

> Use an ExecutorService for repair commands instead of new Thread(..).start()
> 
>
> Key: CASSANDRA-13594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13594
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> Currently when starting a new repair, we create a new Thread and start it 
> immediately
> It would be nice to be able to 1) limit the number of threads and 2) reject 
> starting new repair commands if we are already running too many.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13067) Integer overflows with file system size reported by Amazon Elastic File System (EFS)

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-13067:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Committed into 2.2 at 270f690ff6047cc3e797a3f34b7efa26e7232183 and merged into 
3.0, 3.11 and trunk

> Integer overflows with file system size reported by Amazon Elastic File 
> System (EFS)
> 
>
> Key: CASSANDRA-13067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13067
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra in OpenShift running on Amazon EC2 instance 
> with EFS mounted for data
>Reporter: Michael Hanselmann
>Assignee: Benjamin Lerer
> Attachments: 0001-Handle-exabyte-sized-filesystems.patch
>
>
> When not explicitly configured Cassandra uses 
> [{{nio.FileStore.getTotalSpace}}|https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileStore.html]
>  to determine the total amount of available space in order to [calculate the 
> preferred commit log 
> size|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L553].
>  [Amazon EFS|https://aws.amazon.com/efs/] instances report a filesystem size 
> of 8 EiB when empty. [{{getTotalSpace}} causes an integer overflow 
> (JDK-8162520)|https://bugs.openjdk.java.net/browse/JDK-8162520] and returns a 
> negative number, resulting in a negative preferred size and causing the 
> checked integer to throw.
> Overriding {{commitlog_total_space_in_mb}} is not sufficient as 
> [{{DataDirectory.getAvailableSpace}}|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/db/Directories.java#L550]
>  makes use of {{nio.FileStore.getUsableSpace}}.
> [AMQ-6441] is a comparable issue in ActiveMQ.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13067) Integer overflows with file system size reported by Amazon Elastic File System (EFS)

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-13067:
---
Fix Version/s: 4.0
   3.11.1
   3.0.15
   2.2.11

> Integer overflows with file system size reported by Amazon Elastic File 
> System (EFS)
> 
>
> Key: CASSANDRA-13067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13067
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra in OpenShift running on Amazon EC2 instance 
> with EFS mounted for data
>Reporter: Michael Hanselmann
>Assignee: Benjamin Lerer
> Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0
>
> Attachments: 0001-Handle-exabyte-sized-filesystems.patch
>
>
> When not explicitly configured Cassandra uses 
> [{{nio.FileStore.getTotalSpace}}|https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileStore.html]
>  to determine the total amount of available space in order to [calculate the 
> preferred commit log 
> size|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L553].
>  [Amazon EFS|https://aws.amazon.com/efs/] instances report a filesystem size 
> of 8 EiB when empty. [{{getTotalSpace}} causes an integer overflow 
> (JDK-8162520)|https://bugs.openjdk.java.net/browse/JDK-8162520] and returns a 
> negative number, resulting in a negative preferred size and causing the 
> checked integer to throw.
> Overriding {{commitlog_total_space_in_mb}} is not sufficient as 
> [{{DataDirectory.getAvailableSpace}}|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/db/Directories.java#L550]
>  makes use of {{nio.FileStore.getUsableSpace}}.
> [AMQ-6441] is a comparable issue in ActiveMQ.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13067) Integer overflows with file system size reported by Amazon Elastic File System (EFS)

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-13067:
---
Component/s: Core

> Integer overflows with file system size reported by Amazon Elastic File 
> System (EFS)
> 
>
> Key: CASSANDRA-13067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13067
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra in OpenShift running on Amazon EC2 instance 
> with EFS mounted for data
>Reporter: Michael Hanselmann
>Assignee: Benjamin Lerer
> Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0
>
> Attachments: 0001-Handle-exabyte-sized-filesystems.patch
>
>
> When not explicitly configured Cassandra uses 
> [{{nio.FileStore.getTotalSpace}}|https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileStore.html]
>  to determine the total amount of available space in order to [calculate the 
> preferred commit log 
> size|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L553].
>  [Amazon EFS|https://aws.amazon.com/efs/] instances report a filesystem size 
> of 8 EiB when empty. [{{getTotalSpace}} causes an integer overflow 
> (JDK-8162520)|https://bugs.openjdk.java.net/browse/JDK-8162520] and returns a 
> negative number, resulting in a negative preferred size and causing the 
> checked integer to throw.
> Overriding {{commitlog_total_space_in_mb}} is not sufficient as 
> [{{DataDirectory.getAvailableSpace}}|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/db/Directories.java#L550]
>  makes use of {{nio.FileStore.getUsableSpace}}.
> [AMQ-6441] is a comparable issue in ActiveMQ.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[04/10] cassandra git commit: Prevent integer overflow on exabyte filesystems

[blerer]

Prevent integer overflow on exabyte filesystems

patch by Matt Wringe and Benjamin Lerer; reviewed by Alex Petrov for 
CASSANDRA-13067


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/270f690f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/270f690f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/270f690f

Branch: refs/heads/trunk
Commit: 270f690ff6047cc3e797a3f34b7efa26e7232183
Parents: 739cd2b
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:51:03 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 16:51:03 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 182 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 36c34a1..f712333 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
  * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 981026d..90a82fe 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -21,7 +21,6 @@ import java.io.File;
 import java.io.IOException;
 import java.net.*;
 import java.nio.file.FileStore;
-import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -56,6 +55,9 @@ import org.apache.cassandra.thrift.ThriftServer;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.memory.*;
 
+import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
+import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+
 public class DatabaseDescriptor
 {
 private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@ -530,7 +532,7 @@ public class DatabaseDescriptor
 try
 {
 // use 1/4 of available space.  See discussion on #10013 and 
#10199
-minSize = 
Ints.checkedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
+minSize = 
Ints.saturatedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
 }
 catch (IOException e)
 {
@@ -576,7 +578,7 @@ public class DatabaseDescriptor
 
 try
 {
-dataFreeBytes += guessFileStore(datadir).getUnallocatedSpace();
+dataFreeBytes = saturatedSum(dataFreeBytes, 
guessFileStore(datadir).getUnallocatedSpace());
 }
 catch (IOException e)
 {
@@ -585,9 +587,9 @@ public class DatabaseDescriptor
datadir), e);
 }
 }
-if (dataFreeBytes < 64L * 1024 * 1048576) // 64 GB
+if (dataFreeBytes < 64 * ONE_GB)
 logger.warn("Only {} MB free across all data volumes. Consider 
adding more capacity to your cluster or removing obsolete snapshots",
-dataFreeBytes / 1048576);
+dataFreeBytes / ONE_MB);
 
 
 if (conf.commitlog_directory.equals(conf.saved_caches_directory))
@@ -697,6 +699,20 @@ public class DatabaseDescriptor
 throw new 
ConfigurationException("otc_coalescing_enough_coalesced_messages must be 
positive", false);
 }
 
+/**
+ * Computes the sum of the 2 specified positive values returning {@code 
Long.MAX_VALUE} if the sum overflow.
+ *
+ * @param left the left operand
+ * @param right the right operand
+ * @return the sum of the 2 specified positive values of {@code 
Long.MAX_VALUE} if the sum overflow.
+ */
+private static long saturatedSum(long left, long right)
+{
+assert left >= 0 && right >= 0;
+long sum = left + right;
+return sum < 0 ? Long.MAX_VALUE : sum;
+}
+
 private static FileStore 

[09/10] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11

[blerer]

Merge branch cassandra-3.0 into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/303dba65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/303dba65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/303dba65

Branch: refs/heads/trunk
Commit: 303dba6504ba069b5ea92ee18a47d3ba87c1563e
Parents: 47a2839 1a70ded
Author: Benjamin Lerer 
Authored: Tue Aug 8 17:09:06 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:11:24 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  26 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 191 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/303dba65/CHANGES.txt
--
diff --cc CHANGES.txt
index b778df6,1f42c70..145a746
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -17,9 -13,11 +17,10 @@@ Merged from 3.0
   * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
   * Set test.runners based on cores and memory size (CASSANDRA-13078)
   * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 - * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
   * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
   * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 - Merged from 2.2:
 +Merged from 2.2:
+  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/303dba65/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --cc src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 87b388e,aba7617..ad43565
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@@ -19,10 -19,8 +19,9 @@@ package org.apache.cassandra.config
  
  import java.io.File;
  import java.io.IOException;
 +import java.lang.reflect.Constructor;
  import java.net.*;
  import java.nio.file.FileStore;
- import java.nio.file.Files;
  import java.nio.file.NoSuchFileException;
  import java.nio.file.Path;
  import java.nio.file.Paths;
@@@ -46,25 -40,24 +45,27 @@@ import org.apache.cassandra.config.Conf
  import org.apache.cassandra.dht.IPartitioner;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.io.FSWriteError;
 -import org.apache.cassandra.io.sstable.format.SSTableFormat;
 +import org.apache.cassandra.io.util.DiskOptimizationStrategy;
  import org.apache.cassandra.io.util.FileUtils;
 -import org.apache.cassandra.locator.*;
 -import org.apache.cassandra.net.MessagingService;
 +import org.apache.cassandra.io.util.SpinningDiskOptimizationStrategy;
 +import org.apache.cassandra.io.util.SsdDiskOptimizationStrategy;
 +import org.apache.cassandra.locator.DynamicEndpointSnitch;
 +import org.apache.cassandra.locator.EndpointSnitchInfo;
 +import org.apache.cassandra.locator.IEndpointSnitch;
 +import org.apache.cassandra.locator.SeedProvider;
 +import org.apache.cassandra.net.BackPressureStrategy;
 +import org.apache.cassandra.net.RateBasedBackPressure;
  import org.apache.cassandra.scheduler.IRequestScheduler;
  import org.apache.cassandra.scheduler.NoScheduler;
 -import org.apache.cassandra.service.CacheService;
 -import org.apache.cassandra.thrift.ThriftServer;
 +import org.apache.cassandra.security.EncryptionContext;
 +import org.apache.cassandra.service.CacheService.CacheType;
 +import org.apache.cassandra.thrift.ThriftServer.ThriftServerType;
  import org.apache.cassandra.utils.FBUtilities;
 -import org.apache.cassandra.utils.memory.*;
 +
 +import org.apache.commons.lang3.StringUtils;
  
+ import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
 -import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+ 
  public class DatabaseDescriptor
  {
  private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@@ -429,53 -443,96 +430,53 @@@
  if (conf.native_transport_max_frame_size_in_mb <= 0)
  throw new 
ConfigurationException("native_transport_max_frame_size_in_mb must be positive, 
but was " + conf.native_transport_max_frame_size_in_mb, false);
  
 -// fail early 

[05/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a70dede
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a70dede
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a70dede

Branch: refs/heads/cassandra-3.11
Commit: 1a70dede3a33a2e8bf107f44139580b4ace1c9ff
Parents: 3960260 270f690
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:55:26 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:08:19 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/CHANGES.txt
--
diff --cc CHANGES.txt
index 1525289,f712333..1f42c70
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,22 -1,5 +1,23 @@@
 -2.2.11
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 +3.0.15
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + Merged from 2.2:
++ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/db/Directories.java
--
diff --cc src/java/org/apache/cassandra/db/Directories.java
index 68aa6be,fa76b61..ae7700c
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@@ -523,26 -482,7 +523,25 @@@ public class Directorie
  
  public long getAvailableSpace()
  {
- long availableSpace = location.getUsableSpace() - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 -return FileUtils.getUsableSpace(location);
++long availableSpace = FileUtils.getUsableSpace(location) - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 +return availableSpace > 0 ? availableSpace : 0;
 +}
 +
 +@Override
 +public boolean equals(Object o)
 +{
 +if (this == o) return true;
 +if (o == null || getClass() != o.getClass()) return false;
 +
 +DataDirectory that = (DataDirectory) o;
 +
 +return location.equals(that.location);
- 
 +}
 +
 +@Override
 +public int hashCode()
 +{
 +return location.hashCode();
  }
  }
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/io/util/FileUtils.java
--
diff --cc src/java/org/apache/cassandra/io/util/FileUtils.java
index 

[02/10] cassandra git commit: Prevent integer overflow on exabyte filesystems

[blerer]

Prevent integer overflow on exabyte filesystems

patch by Matt Wringe and Benjamin Lerer; reviewed by Alex Petrov for 
CASSANDRA-13067


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/270f690f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/270f690f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/270f690f

Branch: refs/heads/cassandra-3.0
Commit: 270f690ff6047cc3e797a3f34b7efa26e7232183
Parents: 739cd2b
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:51:03 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 16:51:03 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 182 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 36c34a1..f712333 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
  * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 981026d..90a82fe 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -21,7 +21,6 @@ import java.io.File;
 import java.io.IOException;
 import java.net.*;
 import java.nio.file.FileStore;
-import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -56,6 +55,9 @@ import org.apache.cassandra.thrift.ThriftServer;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.memory.*;
 
+import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
+import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+
 public class DatabaseDescriptor
 {
 private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@ -530,7 +532,7 @@ public class DatabaseDescriptor
 try
 {
 // use 1/4 of available space.  See discussion on #10013 and 
#10199
-minSize = 
Ints.checkedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
+minSize = 
Ints.saturatedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
 }
 catch (IOException e)
 {
@@ -576,7 +578,7 @@ public class DatabaseDescriptor
 
 try
 {
-dataFreeBytes += guessFileStore(datadir).getUnallocatedSpace();
+dataFreeBytes = saturatedSum(dataFreeBytes, 
guessFileStore(datadir).getUnallocatedSpace());
 }
 catch (IOException e)
 {
@@ -585,9 +587,9 @@ public class DatabaseDescriptor
datadir), e);
 }
 }
-if (dataFreeBytes < 64L * 1024 * 1048576) // 64 GB
+if (dataFreeBytes < 64 * ONE_GB)
 logger.warn("Only {} MB free across all data volumes. Consider 
adding more capacity to your cluster or removing obsolete snapshots",
-dataFreeBytes / 1048576);
+dataFreeBytes / ONE_MB);
 
 
 if (conf.commitlog_directory.equals(conf.saved_caches_directory))
@@ -697,6 +699,20 @@ public class DatabaseDescriptor
 throw new 
ConfigurationException("otc_coalescing_enough_coalesced_messages must be 
positive", false);
 }
 
+/**
+ * Computes the sum of the 2 specified positive values returning {@code 
Long.MAX_VALUE} if the sum overflow.
+ *
+ * @param left the left operand
+ * @param right the right operand
+ * @return the sum of the 2 specified positive values of {@code 
Long.MAX_VALUE} if the sum overflow.
+ */
+private static long saturatedSum(long left, long right)
+{
+assert left >= 0 && right >= 0;
+long sum = left + right;
+return sum < 0 ? Long.MAX_VALUE : sum;
+}
+
 private static 

[07/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a70dede
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a70dede
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a70dede

Branch: refs/heads/trunk
Commit: 1a70dede3a33a2e8bf107f44139580b4ace1c9ff
Parents: 3960260 270f690
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:55:26 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:08:19 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/CHANGES.txt
--
diff --cc CHANGES.txt
index 1525289,f712333..1f42c70
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,22 -1,5 +1,23 @@@
 -2.2.11
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 +3.0.15
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + Merged from 2.2:
++ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/db/Directories.java
--
diff --cc src/java/org/apache/cassandra/db/Directories.java
index 68aa6be,fa76b61..ae7700c
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@@ -523,26 -482,7 +523,25 @@@ public class Directorie
  
  public long getAvailableSpace()
  {
- long availableSpace = location.getUsableSpace() - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 -return FileUtils.getUsableSpace(location);
++long availableSpace = FileUtils.getUsableSpace(location) - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 +return availableSpace > 0 ? availableSpace : 0;
 +}
 +
 +@Override
 +public boolean equals(Object o)
 +{
 +if (this == o) return true;
 +if (o == null || getClass() != o.getClass()) return false;
 +
 +DataDirectory that = (DataDirectory) o;
 +
 +return location.equals(that.location);
- 
 +}
 +
 +@Override
 +public int hashCode()
 +{
 +return location.hashCode();
  }
  }
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/io/util/FileUtils.java
--
diff --cc src/java/org/apache/cassandra/io/util/FileUtils.java
index 0bfbbb1,bf0fae5..80df67b

[08/10] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11

[blerer]

Merge branch cassandra-3.0 into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/303dba65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/303dba65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/303dba65

Branch: refs/heads/cassandra-3.11
Commit: 303dba6504ba069b5ea92ee18a47d3ba87c1563e
Parents: 47a2839 1a70ded
Author: Benjamin Lerer 
Authored: Tue Aug 8 17:09:06 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:11:24 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  26 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 191 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/303dba65/CHANGES.txt
--
diff --cc CHANGES.txt
index b778df6,1f42c70..145a746
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -17,9 -13,11 +17,10 @@@ Merged from 3.0
   * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
   * Set test.runners based on cores and memory size (CASSANDRA-13078)
   * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 - * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
   * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
   * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 - Merged from 2.2:
 +Merged from 2.2:
+  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/303dba65/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --cc src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 87b388e,aba7617..ad43565
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@@ -19,10 -19,8 +19,9 @@@ package org.apache.cassandra.config
  
  import java.io.File;
  import java.io.IOException;
 +import java.lang.reflect.Constructor;
  import java.net.*;
  import java.nio.file.FileStore;
- import java.nio.file.Files;
  import java.nio.file.NoSuchFileException;
  import java.nio.file.Path;
  import java.nio.file.Paths;
@@@ -46,25 -40,24 +45,27 @@@ import org.apache.cassandra.config.Conf
  import org.apache.cassandra.dht.IPartitioner;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.io.FSWriteError;
 -import org.apache.cassandra.io.sstable.format.SSTableFormat;
 +import org.apache.cassandra.io.util.DiskOptimizationStrategy;
  import org.apache.cassandra.io.util.FileUtils;
 -import org.apache.cassandra.locator.*;
 -import org.apache.cassandra.net.MessagingService;
 +import org.apache.cassandra.io.util.SpinningDiskOptimizationStrategy;
 +import org.apache.cassandra.io.util.SsdDiskOptimizationStrategy;
 +import org.apache.cassandra.locator.DynamicEndpointSnitch;
 +import org.apache.cassandra.locator.EndpointSnitchInfo;
 +import org.apache.cassandra.locator.IEndpointSnitch;
 +import org.apache.cassandra.locator.SeedProvider;
 +import org.apache.cassandra.net.BackPressureStrategy;
 +import org.apache.cassandra.net.RateBasedBackPressure;
  import org.apache.cassandra.scheduler.IRequestScheduler;
  import org.apache.cassandra.scheduler.NoScheduler;
 -import org.apache.cassandra.service.CacheService;
 -import org.apache.cassandra.thrift.ThriftServer;
 +import org.apache.cassandra.security.EncryptionContext;
 +import org.apache.cassandra.service.CacheService.CacheType;
 +import org.apache.cassandra.thrift.ThriftServer.ThriftServerType;
  import org.apache.cassandra.utils.FBUtilities;
 -import org.apache.cassandra.utils.memory.*;
 +
 +import org.apache.commons.lang3.StringUtils;
  
+ import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
 -import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+ 
  public class DatabaseDescriptor
  {
  private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@@ -429,53 -443,96 +430,53 @@@
  if (conf.native_transport_max_frame_size_in_mb <= 0)
  throw new 
ConfigurationException("native_transport_max_frame_size_in_mb must be positive, 
but was " + conf.native_transport_max_frame_size_in_mb, false);
  
 -// 

[10/10] cassandra git commit: Merge branch cassandra-3.11 into trunk

[blerer]

Merge branch cassandra-3.11 into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0020629
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0020629
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0020629

Branch: refs/heads/trunk
Commit: c00206297be451077034daf318eb8912825e135a
Parents: cad9416 303dba6
Author: Benjamin Lerer 
Authored: Tue Aug 8 17:13:18 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:13:30 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  26 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 191 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0020629/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0020629/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0020629/src/java/org/apache/cassandra/db/Directories.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0020629/src/java/org/apache/cassandra/io/util/FileUtils.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[06/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a70dede
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a70dede
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a70dede

Branch: refs/heads/cassandra-3.0
Commit: 1a70dede3a33a2e8bf107f44139580b4ace1c9ff
Parents: 3960260 270f690
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:55:26 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 17:08:19 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   3 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 181 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/CHANGES.txt
--
diff --cc CHANGES.txt
index 1525289,f712333..1f42c70
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,22 -1,5 +1,23 @@@
 -2.2.11
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 +3.0.15
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + Merged from 2.2:
++ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/db/Directories.java
--
diff --cc src/java/org/apache/cassandra/db/Directories.java
index 68aa6be,fa76b61..ae7700c
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@@ -523,26 -482,7 +523,25 @@@ public class Directorie
  
  public long getAvailableSpace()
  {
- long availableSpace = location.getUsableSpace() - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 -return FileUtils.getUsableSpace(location);
++long availableSpace = FileUtils.getUsableSpace(location) - 
DatabaseDescriptor.getMinFreeSpacePerDriveInBytes();
 +return availableSpace > 0 ? availableSpace : 0;
 +}
 +
 +@Override
 +public boolean equals(Object o)
 +{
 +if (this == o) return true;
 +if (o == null || getClass() != o.getClass()) return false;
 +
 +DataDirectory that = (DataDirectory) o;
 +
 +return location.equals(that.location);
- 
 +}
 +
 +@Override
 +public int hashCode()
 +{
 +return location.hashCode();
  }
  }
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a70dede/src/java/org/apache/cassandra/io/util/FileUtils.java
--
diff --cc src/java/org/apache/cassandra/io/util/FileUtils.java
index 

[03/10] cassandra git commit: Prevent integer overflow on exabyte filesystems

[blerer]

Prevent integer overflow on exabyte filesystems

patch by Matt Wringe and Benjamin Lerer; reviewed by Alex Petrov for 
CASSANDRA-13067


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/270f690f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/270f690f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/270f690f

Branch: refs/heads/cassandra-3.11
Commit: 270f690ff6047cc3e797a3f34b7efa26e7232183
Parents: 739cd2b
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:51:03 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 16:51:03 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 182 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 36c34a1..f712333 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
  * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 981026d..90a82fe 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -21,7 +21,6 @@ import java.io.File;
 import java.io.IOException;
 import java.net.*;
 import java.nio.file.FileStore;
-import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -56,6 +55,9 @@ import org.apache.cassandra.thrift.ThriftServer;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.memory.*;
 
+import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
+import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+
 public class DatabaseDescriptor
 {
 private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@ -530,7 +532,7 @@ public class DatabaseDescriptor
 try
 {
 // use 1/4 of available space.  See discussion on #10013 and 
#10199
-minSize = 
Ints.checkedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
+minSize = 
Ints.saturatedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
 }
 catch (IOException e)
 {
@@ -576,7 +578,7 @@ public class DatabaseDescriptor
 
 try
 {
-dataFreeBytes += guessFileStore(datadir).getUnallocatedSpace();
+dataFreeBytes = saturatedSum(dataFreeBytes, 
guessFileStore(datadir).getUnallocatedSpace());
 }
 catch (IOException e)
 {
@@ -585,9 +587,9 @@ public class DatabaseDescriptor
datadir), e);
 }
 }
-if (dataFreeBytes < 64L * 1024 * 1048576) // 64 GB
+if (dataFreeBytes < 64 * ONE_GB)
 logger.warn("Only {} MB free across all data volumes. Consider 
adding more capacity to your cluster or removing obsolete snapshots",
-dataFreeBytes / 1048576);
+dataFreeBytes / ONE_MB);
 
 
 if (conf.commitlog_directory.equals(conf.saved_caches_directory))
@@ -697,6 +699,20 @@ public class DatabaseDescriptor
 throw new 
ConfigurationException("otc_coalescing_enough_coalesced_messages must be 
positive", false);
 }
 
+/**
+ * Computes the sum of the 2 specified positive values returning {@code 
Long.MAX_VALUE} if the sum overflow.
+ *
+ * @param left the left operand
+ * @param right the right operand
+ * @return the sum of the 2 specified positive values of {@code 
Long.MAX_VALUE} if the sum overflow.
+ */
+private static long saturatedSum(long left, long right)
+{
+assert left >= 0 && right >= 0;
+long sum = left + right;
+return sum < 0 ? Long.MAX_VALUE : sum;
+}
+
 private static 

[01/10] cassandra git commit: Prevent integer overflow on exabyte filesystems

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 739cd2bc2 -> 270f690ff
  refs/heads/cassandra-3.0 396026047 -> 1a70dede3
  refs/heads/cassandra-3.11 47a2839bf -> 303dba650
  refs/heads/trunk cad941653 -> c00206297


Prevent integer overflow on exabyte filesystems

patch by Matt Wringe and Benjamin Lerer; reviewed by Alex Petrov for 
CASSANDRA-13067


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/270f690f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/270f690f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/270f690f

Branch: refs/heads/cassandra-2.2
Commit: 270f690ff6047cc3e797a3f34b7efa26e7232183
Parents: 739cd2b
Author: Benjamin Lerer 
Authored: Tue Aug 8 16:51:03 2017 +0200
Committer: Benjamin Lerer 
Committed: Tue Aug 8 16:51:03 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/config/DatabaseDescriptor.java|  28 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../org/apache/cassandra/io/util/FileUtils.java | 182 +--
 4 files changed, 193 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 36c34a1..f712333 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
  * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223)
  * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
  * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/270f690f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 981026d..90a82fe 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -21,7 +21,6 @@ import java.io.File;
 import java.io.IOException;
 import java.net.*;
 import java.nio.file.FileStore;
-import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -56,6 +55,9 @@ import org.apache.cassandra.thrift.ThriftServer;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.memory.*;
 
+import static org.apache.cassandra.io.util.FileUtils.ONE_GB;
+import static org.apache.cassandra.io.util.FileUtils.ONE_MB;
+
 public class DatabaseDescriptor
 {
 private static final Logger logger = 
LoggerFactory.getLogger(DatabaseDescriptor.class);
@@ -530,7 +532,7 @@ public class DatabaseDescriptor
 try
 {
 // use 1/4 of available space.  See discussion on #10013 and 
#10199
-minSize = 
Ints.checkedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
+minSize = 
Ints.saturatedCast((guessFileStore(conf.commitlog_directory).getTotalSpace() / 
1048576) / 4);
 }
 catch (IOException e)
 {
@@ -576,7 +578,7 @@ public class DatabaseDescriptor
 
 try
 {
-dataFreeBytes += guessFileStore(datadir).getUnallocatedSpace();
+dataFreeBytes = saturatedSum(dataFreeBytes, 
guessFileStore(datadir).getUnallocatedSpace());
 }
 catch (IOException e)
 {
@@ -585,9 +587,9 @@ public class DatabaseDescriptor
datadir), e);
 }
 }
-if (dataFreeBytes < 64L * 1024 * 1048576) // 64 GB
+if (dataFreeBytes < 64 * ONE_GB)
 logger.warn("Only {} MB free across all data volumes. Consider 
adding more capacity to your cluster or removing obsolete snapshots",
-dataFreeBytes / 1048576);
+dataFreeBytes / ONE_MB);
 
 
 if (conf.commitlog_directory.equals(conf.saved_caches_directory))
@@ -697,6 +699,20 @@ public class DatabaseDescriptor
 throw new 
ConfigurationException("otc_coalescing_enough_coalesced_messages must be 
positive", false);
 }
 
+/**
+ * Computes the sum of the 2 specified positive values returning {@code 
Long.MAX_VALUE} if the sum overflow.
+ *
+ * @param left the left operand
+ * @param right the right operand
+ * @return the sum of the 2 specified positive values of {@code 
Long.MAX_VALUE} if the sum 

[jira] [Commented] (CASSANDRA-13594) Use an ExecutorService for repair commands instead of new Thread(..).start()

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118439#comment-16118439
 ] 

Ariel Weisberg commented on CASSANDRA-13594:


I've noticed that short_read test seems to time out quite a bit. I'll run it 
locally a bit and see if it reproduces. I think I did before with no luck. This 
an issue might only appear in Apache Jenkins.

> Use an ExecutorService for repair commands instead of new Thread(..).start()
> 
>
> Key: CASSANDRA-13594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13594
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> Currently when starting a new repair, we create a new Thread and start it 
> immediately
> It would be nice to be able to 1) limit the number of threads and 2) reject 
> starting new repair commands if we are already running too many.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13573) ColumnMetadata.cellValueType() doesn't return correct type for non-frozen collection

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118435#comment-16118435
 ] 

ZhaoYang commented on CASSANDRA-13573:
--

Thank you both for your review

> ColumnMetadata.cellValueType() doesn't return correct type for non-frozen 
> collection
> 
>
> Key: CASSANDRA-13573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13573
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, CQL, Materialized Views, Tools
>Reporter: Stefano Ortolani
>Assignee: ZhaoYang
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Schema and data"
> {noformat}
> CREATE TABLE ks.cf (
> hash blob,
> report_id timeuuid,
> subject_ids frozen,
> PRIMARY KEY (hash, report_id)
> ) WITH CLUSTERING ORDER BY (report_id DESC);
> INSERT INTO ks.cf (hash, report_id, subject_ids) VALUES (0x1213, now(), 
> {1,2,4,5});
> {noformat}
> sstabledump output is:
> {noformat}
> sstabledump mc-1-big-Data.db 
> [
>   {
> "partition" : {
>   "key" : [ "1213" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 16,
> "clustering" : [ "ec01eed0-49d9-11e7-b39a-97a96f529c02" ],
> "liveness_info" : { "tstamp" : "2017-06-05T10:29:57.434856Z" },
> "cells" : [
>   { "name" : "subject_ids", "value" : "" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> While the values are really there:
> {noformat}
> cqlsh:ks> select * from cf ;
>  hash   | report_id| subject_ids
> +--+-
>  0x1213 | 02bafff0-49d9-11e7-b39a-97a96f529c02 |   {1, 2, 4}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13573) ColumnMetadata.cellValueType() doesn't return correct type for non-frozen collection

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña updated CASSANDRA-13573:
--
   Resolution: Fixed
Fix Version/s: 4.x
   3.11.x
   3.0.x
   Status: Resolved  (was: Ready to Commit)

> ColumnMetadata.cellValueType() doesn't return correct type for non-frozen 
> collection
> 
>
> Key: CASSANDRA-13573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13573
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, CQL, Materialized Views, Tools
>Reporter: Stefano Ortolani
>Assignee: ZhaoYang
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Schema and data"
> {noformat}
> CREATE TABLE ks.cf (
> hash blob,
> report_id timeuuid,
> subject_ids frozen,
> PRIMARY KEY (hash, report_id)
> ) WITH CLUSTERING ORDER BY (report_id DESC);
> INSERT INTO ks.cf (hash, report_id, subject_ids) VALUES (0x1213, now(), 
> {1,2,4,5});
> {noformat}
> sstabledump output is:
> {noformat}
> sstabledump mc-1-big-Data.db 
> [
>   {
> "partition" : {
>   "key" : [ "1213" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 16,
> "clustering" : [ "ec01eed0-49d9-11e7-b39a-97a96f529c02" ],
> "liveness_info" : { "tstamp" : "2017-06-05T10:29:57.434856Z" },
> "cells" : [
>   { "name" : "subject_ids", "value" : "" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> While the values are really there:
> {noformat}
> cqlsh:ks> select * from cf ;
>  hash   | report_id| subject_ids
> +--+-
>  0x1213 | 02bafff0-49d9-11e7-b39a-97a96f529c02 |   {1, 2, 4}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13573) ColumnMetadata.cellValueType() doesn't return correct type for non-frozen collection

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118425#comment-16118425
 ] 

Andrés de la Peña commented on CASSANDRA-13573:
---

Committed to 3.0 as 
[3960260472fcd4e0243f62cc813992f1365197c6|https://github.com/apache/cassandra/commit/3960260472fcd4e0243f62cc813992f1365197c6]
 and merged into 3.11 and trunk.

Dtest committed to master as 
[959208749d70e5808aec144e87b73e90d56a7f91|https://github.com/apache/cassandra-dtest/commit/959208749d70e5808aec144e87b73e90d56a7f91]

> ColumnMetadata.cellValueType() doesn't return correct type for non-frozen 
> collection
> 
>
> Key: CASSANDRA-13573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13573
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, CQL, Materialized Views, Tools
>Reporter: Stefano Ortolani
>Assignee: ZhaoYang
>
> Schema and data"
> {noformat}
> CREATE TABLE ks.cf (
> hash blob,
> report_id timeuuid,
> subject_ids frozen,
> PRIMARY KEY (hash, report_id)
> ) WITH CLUSTERING ORDER BY (report_id DESC);
> INSERT INTO ks.cf (hash, report_id, subject_ids) VALUES (0x1213, now(), 
> {1,2,4,5});
> {noformat}
> sstabledump output is:
> {noformat}
> sstabledump mc-1-big-Data.db 
> [
>   {
> "partition" : {
>   "key" : [ "1213" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 16,
> "clustering" : [ "ec01eed0-49d9-11e7-b39a-97a96f529c02" ],
> "liveness_info" : { "tstamp" : "2017-06-05T10:29:57.434856Z" },
> "cells" : [
>   { "name" : "subject_ids", "value" : "" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> While the values are really there:
> {noformat}
> cqlsh:ks> select * from cf ;
>  hash   | report_id| subject_ids
> +--+-
>  0x1213 | 02bafff0-49d9-11e7-b39a-97a96f529c02 |   {1, 2, 4}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error

[Eric Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118421#comment-16118421
 ] 

Eric Evans commented on CASSANDRA-13006:


I think this behavior (invoking {{jmap}} on OOM) is a pretty serious violation 
to the Element of Least-surprise.  We already provide mechanisms for passing 
arguments to the JVM, and TTBMK, all of them provide some means for dropping a 
heap dump on out-of-memory.

It definitely caught me be surprise.  We carried over 
{{-XX:+HeapDumpOnOutOfMemoryError}} from our 2.2.x environment, only to have 
Cassandra and the JVM racing to create a dump of the same name.

Additionally, something about all of this is buggy, because on more than one 
occasion we've had Cassandra fork-bombing {{jmap}} processes

{noformat}
● cassandra-b.service - distributed storage system for structured data
   Loaded: loaded (/lib/systemd/system/cassandra-b.service; static)
   Active: active (running) since Sat 2017-08-05 22:32:07 UTC; 23h ago
 Main PID: 25025 (java)
   CGroup: /system.slice/cassandra-b.service
   ├─ 9213 jmap -histo 25025
   ├─ 9214 jmap -histo 25025
   ├─ 9284 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9285 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9388 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9453 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9519 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9520 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9733 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9735 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─ 9736 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14835 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14836 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14837 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14839 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14841 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─14844 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18932 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18933 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18934 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18935 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18936 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18937 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18938 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18939 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18940 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18942 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18943 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18944 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   ├─18945 jmap 
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
   [ ... ]
{noformat}

IMO, the sanest strategy here would be to leave the creation of heap dumps to 
the JVM.



> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by 
> default if the process encountered an OOM error. These heap dumps are stored 
> in the Apache Cassandra home directory unless configured otherwise (see 
> [Cassandra Support 
> 

[8/8] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[adelapena]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cad94165
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cad94165
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cad94165

Branch: refs/heads/trunk
Commit: cad941653b59c4609ba2900cda86dc366d3ccd38
Parents: 6dfd11c 47a2839
Author: Andrés de la Peña 
Authored: Tue Aug 8 15:27:15 2017 +0100
Committer: Andrés de la Peña 
Committed: Tue Aug 8 15:27:15 2017 +0100

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/rows/AbstractCell.java  |  2 +-
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/schema/ColumnMetadata.java | 21 ++---
 .../apache/cassandra/tools/JsonTransformer.java | 32 +--
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 .../cassandra/index/sasi/SASIIndexTest.java | 14 +
 7 files changed, 96 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cad94165/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cad94165/src/java/org/apache/cassandra/db/rows/AbstractCell.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cad94165/src/java/org/apache/cassandra/db/rows/BTreeRow.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cad94165/src/java/org/apache/cassandra/schema/ColumnMetadata.java
--
diff --cc src/java/org/apache/cassandra/schema/ColumnMetadata.java
index ea80708,000..3b55c03
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/schema/ColumnMetadata.java
+++ b/src/java/org/apache/cassandra/schema/ColumnMetadata.java
@@@ -1,626 -1,0 +1,639 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.schema;
 +
 +import java.nio.ByteBuffer;
 +import java.util.*;
 +import java.util.function.Predicate;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import com.google.common.base.MoreObjects;
 +import com.google.common.base.Objects;
 +import com.google.common.collect.Collections2;
 +
 +import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.selection.Selectable;
 +import org.apache.cassandra.cql3.selection.Selector;
 +import org.apache.cassandra.cql3.selection.SimpleSelector;
 +import org.apache.cassandra.db.rows.*;
 +import org.apache.cassandra.db.marshal.*;
 +import org.apache.cassandra.exceptions.InvalidRequestException;
 +import org.apache.cassandra.serializers.MarshalException;
 +import org.apache.cassandra.utils.ByteBufferUtil;
 +import org.github.jamm.Unmetered;
 +
 +@Unmetered
 +public final class ColumnMetadata extends ColumnSpecification implements 
Selectable, Comparable
 +{
 +public static final Comparator asymmetricColumnDataComparator =
 +(a, b) -> ((ColumnData) a).column().compareTo((ColumnMetadata) b);
 +
 +public static final int NO_POSITION = -1;
 +
 +public enum ClusteringOrder
 +{
 +ASC, DESC, NONE
 +}
 +
 +/**
 + * The type of CQL3 column this definition represents.
 + * There is 4 main type of CQL3 columns: those parts of the partition key,
 + * those parts of the clustering columns and amongst the others, regular 
and
 + * static ones.
 + *
 + * IMPORTANT: this enum is serialized as toString() and deserialized by 
calling
 + * Kind.valueOf(), so do not override toString() or rename existing 
values.
 + */
 +public enum Kind
 +{
 +// NOTE: if adding a new type, must modify comparisonOrder
 +PARTITION_KEY,
 +CLUSTERING,
 +REGULAR,
 +STATIC;
 +
 +public boolean 

[5/8] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[adelapena]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47a2839b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47a2839b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47a2839b

Branch: refs/heads/trunk
Commit: 47a2839bf094a7fa9bad6de140fba486756c49bf
Parents: 15abe2d 3960260
Author: Andrés de la Peña 
Authored: Tue Aug 8 15:01:04 2017 +0100
Committer: Andrés de la Peña 
Committed: Tue Aug 8 15:01:04 2017 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/config/ColumnDefinition.java  | 19 ---
 .../apache/cassandra/db/rows/AbstractCell.java  |  2 +-
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/tools/JsonTransformer.java | 31 --
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 .../cassandra/index/sasi/SASIIndexTest.java | 14 +
 7 files changed, 94 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/CHANGES.txt
--
diff --cc CHANGES.txt
index b8c4bde,1525289..b778df6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.15
 +3.11.1
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
   * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
   * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
   * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --cc src/java/org/apache/cassandra/config/ColumnDefinition.java
index ea508d2,6a0f530..159ea0c
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@@ -446,197 -398,17 +448,206 @@@ public class ColumnDefinition extends C
   */
  public AbstractType cellValueType()
  {
- return type instanceof CollectionType
-  ? ((CollectionType)type).valueComparator()
-  : type;
+ assert !(type instanceof UserType && type.isMultiCell());
+ return type instanceof CollectionType && type.isMultiCell()
+ ? ((CollectionType)type).valueComparator()
+ : type;
+ }
+ 
+ 
+ public boolean isCounterColumn()
+ {
+ if (type instanceof CollectionType) // for thrift
+ return ((CollectionType) type).valueComparator().isCounter();
+ return type.isCounter();
  }
 +
 +public Selector.Factory newSelectorFactory(CFMetaData cfm, 
AbstractType expectedType, List defs, 
VariableSpecifications boundNames) throws InvalidRequestException
 +{
 +return SimpleSelector.newFactory(this, addAndGetIndex(this, defs));
 +}
 +
 +public AbstractType getExactTypeIfKnown(String keyspace)
 +{
 +return type;
 +}
 +
 +/**
 + * Because Thrift-created tables may have a non-text comparator, we 
cannot determine the proper 'key' until
 + * we know the comparator. ColumnDefinition.Raw is a placeholder that can 
be converted to a real ColumnIdentifier
 + * once the comparator is known with prepare(). This should only be used 
with identifiers that are actual
 + * column names. See CASSANDRA-8178 for more background.
 + */
 +public static abstract class Raw extends Selectable.Raw
 +{
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from an unquoted identifier 
string.
 + */
 +public static Raw forUnquoted(String text)
 +{
 +return new Literal(text, false);
 +}
 +
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from a quoted identifier 
string.
 + */
 +public static Raw forQuoted(String text)
 +{
 +return new Literal(text, true);
 +}
 +
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from a pre-existing {@code 
ColumnDefinition}
 + * (useful in the rare cases where we already have the column but need
 + * a {@code 

[4/8] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[adelapena]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
--
diff --cc test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
index a6ce08b,000..e9051b4
mode 100644,00..100644
--- a/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
+++ b/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
@@@ -1,2552 -1,0 +1,2566 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.index.sasi;
 +
 +import java.io.FileWriter;
 +import java.io.Writer;
 +import java.nio.ByteBuffer;
 +import java.nio.file.FileSystems;
 +import java.nio.file.Files;
 +import java.nio.file.Path;
 +import java.nio.file.attribute.BasicFileAttributes;
 +import java.util.*;
 +import java.util.concurrent.ExecutorService;
 +import java.util.concurrent.Executors;
 +import java.util.concurrent.ThreadLocalRandom;
 +import java.util.concurrent.TimeUnit;
 +import java.util.concurrent.atomic.AtomicInteger;
 +
 +import org.apache.cassandra.SchemaLoader;
 +import org.apache.cassandra.config.CFMetaData;
 +import org.apache.cassandra.config.ColumnDefinition;
 +import org.apache.cassandra.index.Index;
 +import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.Term;
 +import org.apache.cassandra.cql3.statements.IndexTarget;
 +import org.apache.cassandra.cql3.statements.SelectStatement;
 +import org.apache.cassandra.db.*;
 +import org.apache.cassandra.db.filter.ColumnFilter;
 +import org.apache.cassandra.db.filter.DataLimits;
 +import org.apache.cassandra.db.filter.RowFilter;
 +import org.apache.cassandra.db.lifecycle.SSTableSet;
 +import org.apache.cassandra.db.marshal.*;
 +import org.apache.cassandra.db.partitions.PartitionUpdate;
 +import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
 +import org.apache.cassandra.db.rows.*;
 +import org.apache.cassandra.dht.IPartitioner;
 +import org.apache.cassandra.dht.Murmur3Partitioner;
 +import org.apache.cassandra.dht.Range;
 +import org.apache.cassandra.exceptions.ConfigurationException;
 +import org.apache.cassandra.exceptions.InvalidRequestException;
 +import org.apache.cassandra.index.sasi.conf.ColumnIndex;
 +import org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder;
 +import org.apache.cassandra.index.sasi.exceptions.TimeQuotaExceededException;
 +import org.apache.cassandra.index.sasi.memory.IndexMemtable;
 +import org.apache.cassandra.index.sasi.plan.QueryController;
 +import org.apache.cassandra.index.sasi.plan.QueryPlan;
 +import org.apache.cassandra.io.sstable.SSTable;
 +import org.apache.cassandra.schema.IndexMetadata;
 +import org.apache.cassandra.schema.KeyspaceMetadata;
 +import org.apache.cassandra.schema.KeyspaceParams;
 +import org.apache.cassandra.schema.Tables;
 +import org.apache.cassandra.serializers.MarshalException;
 +import org.apache.cassandra.serializers.TypeSerializer;
 +import org.apache.cassandra.service.MigrationManager;
 +import org.apache.cassandra.service.QueryState;
 +import org.apache.cassandra.thrift.CqlRow;
 +import org.apache.cassandra.transport.messages.ResultMessage;
 +import org.apache.cassandra.utils.ByteBufferUtil;
 +import org.apache.cassandra.utils.FBUtilities;
 +import org.apache.cassandra.utils.Pair;
 +
 +import com.google.common.collect.Lists;
 +import com.google.common.util.concurrent.Uninterruptibles;
 +
 +import junit.framework.Assert;
 +
 +import org.junit.*;
 +
 +public class SASIIndexTest
 +{
 +private static final IPartitioner PARTITIONER;
 +
 +static {
 +System.setProperty("cassandra.config", "cassandra-murmur.yaml");
 +PARTITIONER = Murmur3Partitioner.instance;
 +}
 +
 +private static final String KS_NAME = "sasi";
 +private static final String CF_NAME = "test_cf";
 +private static final String CLUSTERING_CF_NAME_1 = "clustering_test_cf_1";
 +private static final String CLUSTERING_CF_NAME_2 = "clustering_test_cf_2";
 +private static final String STATIC_CF_NAME = "static_sasi_test_cf";
 +private static final String FTS_CF_NAME = 

[2/8] cassandra git commit: Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to use type.toJsonString()

[adelapena]

Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to 
use type.toJsonString()

patch by Zhao Yang, reviewed by Andres de la Peña for CASSANDRA-13573


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/39602604
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/39602604
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/39602604

Branch: refs/heads/cassandra-3.11
Commit: 3960260472fcd4e0243f62cc813992f1365197c6
Parents: 9186679
Author: Zhao Yang 
Authored: Wed Aug 2 11:58:38 2017 +0800
Committer: Andrés de la Peña 
Committed: Tue Aug 8 14:31:23 2017 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/config/ColumnDefinition.java  | 19 ---
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/db/rows/BufferCell.java|  2 +-
 .../apache/cassandra/tools/JsonTransformer.java |  6 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 6 files changed, 55 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 905a436..1525289 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
  * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
  * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
  * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 34840e3..6a0f530 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -391,13 +391,24 @@ public class ColumnDefinition extends ColumnSpecification 
implements Comparable<
 /**
  * The type of the cell values for cell belonging to this column.
  *
- * This is the same than the column type, except for collections where 
it's the 'valueComparator'
+ * This is the same than the column type, except for non-frozen 
collections where it's the 'valueComparator'
  * of the collection.
+ * 
+ * This method should not be used to get value type of non-frozon UDT.
  */
 public AbstractType cellValueType()
 {
-return type instanceof CollectionType
- ? ((CollectionType)type).valueComparator()
- : type;
+assert !(type instanceof UserType && type.isMultiCell());
+return type instanceof CollectionType && type.isMultiCell()
+? ((CollectionType)type).valueComparator()
+: type;
+}
+
+
+public boolean isCounterColumn()
+{
+if (type instanceof CollectionType) // for thrift
+return ((CollectionType) type).valueComparator().isCounter();
+return type.isCounter();
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BTreeRow.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/BTreeRow.java 
b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
index fda33d6..41dad0a 100644
--- a/src/java/org/apache/cassandra/db/rows/BTreeRow.java
+++ b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
@@ -294,7 +294,7 @@ public class BTreeRow extends AbstractRow
 
 public Row markCounterLocalToBeCleared()
 {
-return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().cellValueType().isCounter()
+return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().isCounterColumn()
 ? 
cd.markCounterLocalToBeCleared()
 : 
cd);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BufferCell.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/BufferCell.java 
b/src/java/org/apache/cassandra/db/rows/BufferCell.java
index e4ad7e6..82ae02c 100644
--- 

[6/8] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[adelapena]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
--
diff --cc test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
index a6ce08b,000..e9051b4
mode 100644,00..100644
--- a/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
+++ b/test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java
@@@ -1,2552 -1,0 +1,2566 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.index.sasi;
 +
 +import java.io.FileWriter;
 +import java.io.Writer;
 +import java.nio.ByteBuffer;
 +import java.nio.file.FileSystems;
 +import java.nio.file.Files;
 +import java.nio.file.Path;
 +import java.nio.file.attribute.BasicFileAttributes;
 +import java.util.*;
 +import java.util.concurrent.ExecutorService;
 +import java.util.concurrent.Executors;
 +import java.util.concurrent.ThreadLocalRandom;
 +import java.util.concurrent.TimeUnit;
 +import java.util.concurrent.atomic.AtomicInteger;
 +
 +import org.apache.cassandra.SchemaLoader;
 +import org.apache.cassandra.config.CFMetaData;
 +import org.apache.cassandra.config.ColumnDefinition;
 +import org.apache.cassandra.index.Index;
 +import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.Term;
 +import org.apache.cassandra.cql3.statements.IndexTarget;
 +import org.apache.cassandra.cql3.statements.SelectStatement;
 +import org.apache.cassandra.db.*;
 +import org.apache.cassandra.db.filter.ColumnFilter;
 +import org.apache.cassandra.db.filter.DataLimits;
 +import org.apache.cassandra.db.filter.RowFilter;
 +import org.apache.cassandra.db.lifecycle.SSTableSet;
 +import org.apache.cassandra.db.marshal.*;
 +import org.apache.cassandra.db.partitions.PartitionUpdate;
 +import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
 +import org.apache.cassandra.db.rows.*;
 +import org.apache.cassandra.dht.IPartitioner;
 +import org.apache.cassandra.dht.Murmur3Partitioner;
 +import org.apache.cassandra.dht.Range;
 +import org.apache.cassandra.exceptions.ConfigurationException;
 +import org.apache.cassandra.exceptions.InvalidRequestException;
 +import org.apache.cassandra.index.sasi.conf.ColumnIndex;
 +import org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder;
 +import org.apache.cassandra.index.sasi.exceptions.TimeQuotaExceededException;
 +import org.apache.cassandra.index.sasi.memory.IndexMemtable;
 +import org.apache.cassandra.index.sasi.plan.QueryController;
 +import org.apache.cassandra.index.sasi.plan.QueryPlan;
 +import org.apache.cassandra.io.sstable.SSTable;
 +import org.apache.cassandra.schema.IndexMetadata;
 +import org.apache.cassandra.schema.KeyspaceMetadata;
 +import org.apache.cassandra.schema.KeyspaceParams;
 +import org.apache.cassandra.schema.Tables;
 +import org.apache.cassandra.serializers.MarshalException;
 +import org.apache.cassandra.serializers.TypeSerializer;
 +import org.apache.cassandra.service.MigrationManager;
 +import org.apache.cassandra.service.QueryState;
 +import org.apache.cassandra.thrift.CqlRow;
 +import org.apache.cassandra.transport.messages.ResultMessage;
 +import org.apache.cassandra.utils.ByteBufferUtil;
 +import org.apache.cassandra.utils.FBUtilities;
 +import org.apache.cassandra.utils.Pair;
 +
 +import com.google.common.collect.Lists;
 +import com.google.common.util.concurrent.Uninterruptibles;
 +
 +import junit.framework.Assert;
 +
 +import org.junit.*;
 +
 +public class SASIIndexTest
 +{
 +private static final IPartitioner PARTITIONER;
 +
 +static {
 +System.setProperty("cassandra.config", "cassandra-murmur.yaml");
 +PARTITIONER = Murmur3Partitioner.instance;
 +}
 +
 +private static final String KS_NAME = "sasi";
 +private static final String CF_NAME = "test_cf";
 +private static final String CLUSTERING_CF_NAME_1 = "clustering_test_cf_1";
 +private static final String CLUSTERING_CF_NAME_2 = "clustering_test_cf_2";
 +private static final String STATIC_CF_NAME = "static_sasi_test_cf";
 +private static final String FTS_CF_NAME = 

[1/8] cassandra git commit: Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to use type.toJsonString()

[adelapena]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 918667929 -> 396026047
  refs/heads/cassandra-3.11 15abe2db9 -> 47a2839bf
  refs/heads/trunk 6dfd11c30 -> cad941653


Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to 
use type.toJsonString()

patch by Zhao Yang, reviewed by Andres de la Peña for CASSANDRA-13573


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/39602604
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/39602604
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/39602604

Branch: refs/heads/cassandra-3.0
Commit: 3960260472fcd4e0243f62cc813992f1365197c6
Parents: 9186679
Author: Zhao Yang 
Authored: Wed Aug 2 11:58:38 2017 +0800
Committer: Andrés de la Peña 
Committed: Tue Aug 8 14:31:23 2017 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/config/ColumnDefinition.java  | 19 ---
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/db/rows/BufferCell.java|  2 +-
 .../apache/cassandra/tools/JsonTransformer.java |  6 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 6 files changed, 55 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 905a436..1525289 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
  * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
  * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
  * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 34840e3..6a0f530 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -391,13 +391,24 @@ public class ColumnDefinition extends ColumnSpecification 
implements Comparable<
 /**
  * The type of the cell values for cell belonging to this column.
  *
- * This is the same than the column type, except for collections where 
it's the 'valueComparator'
+ * This is the same than the column type, except for non-frozen 
collections where it's the 'valueComparator'
  * of the collection.
+ * 
+ * This method should not be used to get value type of non-frozon UDT.
  */
 public AbstractType cellValueType()
 {
-return type instanceof CollectionType
- ? ((CollectionType)type).valueComparator()
- : type;
+assert !(type instanceof UserType && type.isMultiCell());
+return type instanceof CollectionType && type.isMultiCell()
+? ((CollectionType)type).valueComparator()
+: type;
+}
+
+
+public boolean isCounterColumn()
+{
+if (type instanceof CollectionType) // for thrift
+return ((CollectionType) type).valueComparator().isCounter();
+return type.isCounter();
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BTreeRow.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/BTreeRow.java 
b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
index fda33d6..41dad0a 100644
--- a/src/java/org/apache/cassandra/db/rows/BTreeRow.java
+++ b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
@@ -294,7 +294,7 @@ public class BTreeRow extends AbstractRow
 
 public Row markCounterLocalToBeCleared()
 {
-return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().cellValueType().isCounter()
+return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().isCounterColumn()
 ? 
cd.markCounterLocalToBeCleared()
 : 
cd);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BufferCell.java

[3/8] cassandra git commit: Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to use type.toJsonString()

[adelapena]

Fix ColumnMetadata.cellValueType() return type and change sstabledump tool to 
use type.toJsonString()

patch by Zhao Yang, reviewed by Andres de la Peña for CASSANDRA-13573


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/39602604
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/39602604
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/39602604

Branch: refs/heads/trunk
Commit: 3960260472fcd4e0243f62cc813992f1365197c6
Parents: 9186679
Author: Zhao Yang 
Authored: Wed Aug 2 11:58:38 2017 +0800
Committer: Andrés de la Peña 
Committed: Tue Aug 8 14:31:23 2017 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/config/ColumnDefinition.java  | 19 ---
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/db/rows/BufferCell.java|  2 +-
 .../apache/cassandra/tools/JsonTransformer.java |  6 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 6 files changed, 55 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 905a436..1525289 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
  * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
  * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
  * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 34840e3..6a0f530 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -391,13 +391,24 @@ public class ColumnDefinition extends ColumnSpecification 
implements Comparable<
 /**
  * The type of the cell values for cell belonging to this column.
  *
- * This is the same than the column type, except for collections where 
it's the 'valueComparator'
+ * This is the same than the column type, except for non-frozen 
collections where it's the 'valueComparator'
  * of the collection.
+ * 
+ * This method should not be used to get value type of non-frozon UDT.
  */
 public AbstractType cellValueType()
 {
-return type instanceof CollectionType
- ? ((CollectionType)type).valueComparator()
- : type;
+assert !(type instanceof UserType && type.isMultiCell());
+return type instanceof CollectionType && type.isMultiCell()
+? ((CollectionType)type).valueComparator()
+: type;
+}
+
+
+public boolean isCounterColumn()
+{
+if (type instanceof CollectionType) // for thrift
+return ((CollectionType) type).valueComparator().isCounter();
+return type.isCounter();
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BTreeRow.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/BTreeRow.java 
b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
index fda33d6..41dad0a 100644
--- a/src/java/org/apache/cassandra/db/rows/BTreeRow.java
+++ b/src/java/org/apache/cassandra/db/rows/BTreeRow.java
@@ -294,7 +294,7 @@ public class BTreeRow extends AbstractRow
 
 public Row markCounterLocalToBeCleared()
 {
-return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().cellValueType().isCounter()
+return transformAndFilter(primaryKeyLivenessInfo, deletion, (cd) -> 
cd.column().isCounterColumn()
 ? 
cd.markCounterLocalToBeCleared()
 : 
cd);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39602604/src/java/org/apache/cassandra/db/rows/BufferCell.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/BufferCell.java 
b/src/java/org/apache/cassandra/db/rows/BufferCell.java
index e4ad7e6..82ae02c 100644
--- 

[7/8] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[adelapena]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47a2839b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47a2839b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47a2839b

Branch: refs/heads/cassandra-3.11
Commit: 47a2839bf094a7fa9bad6de140fba486756c49bf
Parents: 15abe2d 3960260
Author: Andrés de la Peña 
Authored: Tue Aug 8 15:01:04 2017 +0100
Committer: Andrés de la Peña 
Committed: Tue Aug 8 15:01:04 2017 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/config/ColumnDefinition.java  | 19 ---
 .../apache/cassandra/db/rows/AbstractCell.java  |  2 +-
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  2 +-
 .../apache/cassandra/tools/JsonTransformer.java | 31 --
 .../org/apache/cassandra/cql3/ViewTest.java | 33 
 .../cassandra/index/sasi/SASIIndexTest.java | 14 +
 7 files changed, 94 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/CHANGES.txt
--
diff --cc CHANGES.txt
index b8c4bde,1525289..b778df6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.15
 +3.11.1
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
   * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
   * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
   * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/47a2839b/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --cc src/java/org/apache/cassandra/config/ColumnDefinition.java
index ea508d2,6a0f530..159ea0c
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@@ -446,197 -398,17 +448,206 @@@ public class ColumnDefinition extends C
   */
  public AbstractType cellValueType()
  {
- return type instanceof CollectionType
-  ? ((CollectionType)type).valueComparator()
-  : type;
+ assert !(type instanceof UserType && type.isMultiCell());
+ return type instanceof CollectionType && type.isMultiCell()
+ ? ((CollectionType)type).valueComparator()
+ : type;
+ }
+ 
+ 
+ public boolean isCounterColumn()
+ {
+ if (type instanceof CollectionType) // for thrift
+ return ((CollectionType) type).valueComparator().isCounter();
+ return type.isCounter();
  }
 +
 +public Selector.Factory newSelectorFactory(CFMetaData cfm, 
AbstractType expectedType, List defs, 
VariableSpecifications boundNames) throws InvalidRequestException
 +{
 +return SimpleSelector.newFactory(this, addAndGetIndex(this, defs));
 +}
 +
 +public AbstractType getExactTypeIfKnown(String keyspace)
 +{
 +return type;
 +}
 +
 +/**
 + * Because Thrift-created tables may have a non-text comparator, we 
cannot determine the proper 'key' until
 + * we know the comparator. ColumnDefinition.Raw is a placeholder that can 
be converted to a real ColumnIdentifier
 + * once the comparator is known with prepare(). This should only be used 
with identifiers that are actual
 + * column names. See CASSANDRA-8178 for more background.
 + */
 +public static abstract class Raw extends Selectable.Raw
 +{
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from an unquoted identifier 
string.
 + */
 +public static Raw forUnquoted(String text)
 +{
 +return new Literal(text, false);
 +}
 +
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from a quoted identifier 
string.
 + */
 +public static Raw forQuoted(String text)
 +{
 +return new Literal(text, true);
 +}
 +
 +/**
 + * Creates a {@code ColumnDefinition.Raw} from a pre-existing {@code 
ColumnDefinition}
 + * (useful in the rare cases where we already have the column but need
 + * 

[jira] [Comment Edited] (CASSANDRA-9988) Introduce leaf-only iterator

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118100#comment-16118100
 ] 

Benedict edited comment on CASSANDRA-9988 at 8/8/17 12:27 PM:
--

bq. The original implementation is valid. I think that's another example shows 
the code is not that easy to understand.

Mea culpa, although I think that's more an indictment of my hubris and our 
mutual disinterest.  If fifteen distracted minutes in a JIRA comment window 
procrastinating from real work yields a minor bug, it's probably not *that* 
complicated, and an IDE, test suite and a couple of hours would probably more 
than conquer it.

bq. The graph is the search performance (throughput) for searching target on 
the different positions.

Except that this isn't the same, because the search domain is different with 
different starting positions.  This is also over-counting unrelated test setup 
costs.  One would expect that, with a leaf size of 32, iterating the whole 
contents would be _approx._ 4x faster with exp. search than binary search (and 
5x faster for a single search from the beginning); the fact it is only 3x is 
probably down to those extra costs.  The worst inversion of performance would 
be around 2x (i.e. exp. search being 2x slower than b.search at its worst), 
with logarithmically spaced items - i.e. at position 16, 8, 4, 2, 1.  However, 
in the slowest case for exp. search, we are doing much less work than the 
slowest for binary search.

Of course, with a limit to 32 items the exp. search benefits are much less 
apparent.  If we had a leaf size of 64 it would become much more stark.  With a 
leaf size of 16, it would also become completely irrelevant.




was (Author: benedict):
bq. The original implementation is valid. I think that's another example shows 
the code is not that easy to understand.

Mea culpa, although I think that's more an indictment of my hubris and our 
mutual disinterest.  If fifteen distracted minutes in a JIRA comment window 
procrastinating from real work yields a minor bug, it's probably not *that* 
complicated, and an IDE, test suite and a couple of hours would probably more 
than conquer it.

bq. The graph is the search performance (throughput) for searching target on 
the different positions.

Except that this isn't the same, because the search domain is different with 
different starting positions.  This is also over-counting unrelated test setup 
costs.  One would expect that, with a leaf size of 32, iterating the whole 
contents would be _approx._ 4x faster with exp. search than binary search; the 
fact it is only 3x is probably down to those extra costs.  The worst inversion 
of performance would be around 2x (i.e. exp. search being 2x slower than 
b.search at its worst), with logarithmically spaced items - i.e. at position 
16, 8, 4, 2, 1.  However, in the slowest case for exp. search, we are doing 
much less work than the slowest for binary search.

Of course, with a limit to 32 items the exp. search benefits are much less 
apparent.  If we had a leaf size of 64 it would become much more stark.  With a 
leaf size of 16, it would also become completely irrelevant.



> Introduce leaf-only iterator
> 
>
> Key: CASSANDRA-9988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9988
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Benedict
>Assignee: Jay Zhuang
>Priority: Minor
>  Labels: patch
> Fix For: 4.0
>
> Attachments: 9988-3tests.png, 9988-data.png, 9988-result2.png, 
> 9988-result3.png, 9988-result.png, 9988-test-result3.png, 
> 9988-test-result-expsearch.xlsx, 9988-test-result-raw.png, 
> 9988-test-result.xlsx, 9988-trunk-new.txt, 9988-trunk-new-update.txt, 
> trunk-9988.txt
>
>
> In many cases we have small btrees, small enough to fit in a single leaf 
> page. In this case it _may_ be more efficient to specialise our iterator.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13594) Use an ExecutorService for repair commands instead of new Thread(..).start()

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118249#comment-16118249
 ] 

Marcus Eriksson commented on CASSANDRA-13594:
-

Still trying to get dtests to run successfully, they seem quite broken right now

> Use an ExecutorService for repair commands instead of new Thread(..).start()
> 
>
> Key: CASSANDRA-13594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13594
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> Currently when starting a new repair, we create a new Thread and start it 
> immediately
> It would be nice to be able to 1) limit the number of threads and 2) reject 
> starting new repair commands if we are already running too many.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11483) Enhance sstablemetadata

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-11483:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1, committed!

> Enhance sstablemetadata
> ---
>
> Key: CASSANDRA-11483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11483
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Fix For: 4.0
>
> Attachments: CASSANDRA-11483.txt, CASSANDRA-11483v2.txt, 
> CASSANDRA-11483v3.txt, CASSANDRA-11483v4.txt, CASSANDRA-11483v5.txt, Screen 
> Shot 2016-04-03 at 11.40.32 PM.png
>
>
> sstablemetadata provides quite a bit of useful information but theres a few 
> hiccups I would like to see addressed:
> * Does not use client mode
> * Units are not provided (or anything for that matter). There is data in 
> micros, millis, seconds as durations and timestamps from epoch. But there is 
> no way to tell what one is without a non-trival code dive
> * in general pretty frustrating to parse



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Improve sstablemetadata output

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 9e3483f84 -> 6dfd11c30


Improve sstablemetadata output

Patch by Chris Lohfink; reviewed by marcuse for CASSANDRA-11483


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6dfd11c3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6dfd11c3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6dfd11c3

Branch: refs/heads/trunk
Commit: 6dfd11c30a9c85581b77c93cfcdbef37a5d497c6
Parents: 9e3483f
Author: Chris Lohfink 
Authored: Fri Jul 21 20:04:37 2017 -0700
Committer: Marcus Eriksson 
Committed: Tue Aug 8 13:57:43 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/tools/SSTableExport.java   |  69 +--
 .../cassandra/tools/SSTableMetadataViewer.java  | 600 +++
 src/java/org/apache/cassandra/tools/Util.java   | 282 +
 .../cassandra/utils/EstimatedHistogram.java |  27 +-
 5 files changed, 792 insertions(+), 187 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6dfd11c3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 69b8eaa..0ddcf94 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Improve sstablemetadata output (CASSANDRA-11483)
  * Support for migrating legacy users to roles has been dropped 
(CASSANDRA-13371)
  * Introduce error metrics for repair (CASSANDRA-13387)
  * Refactoring to primitive functional interfaces in AuthCache 
(CASSANDRA-13732)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6dfd11c3/src/java/org/apache/cassandra/tools/SSTableExport.java
--
diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
b/src/java/org/apache/cassandra/tools/SSTableExport.java
index 5d22462..26dae7f 100644
--- a/src/java/org/apache/cassandra/tools/SSTableExport.java
+++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
@@ -19,28 +19,33 @@ package org.apache.cassandra.tools;
 
 import java.io.File;
 import java.io.IOException;
-import java.util.*;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
-import java.util.stream.StreamSupport;
-
-import org.apache.commons.cli.*;
 
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.config.DatabaseDescriptor;
-import org.apache.cassandra.cql3.ColumnIdentifier;
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.PartitionPosition;
-import org.apache.cassandra.db.SerializationHeader;
-import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
-import org.apache.cassandra.dht.*;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.Bounds;
+import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.ISSTableScanner;
 import org.apache.cassandra.io.sstable.KeyIterator;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
 import org.apache.cassandra.io.sstable.metadata.MetadataComponent;
 import org.apache.cassandra.io.sstable.metadata.MetadataType;
 import org.apache.cassandra.schema.TableMetadataRef;
@@ -86,48 +91,6 @@ public class SSTableExport
 }
 
 /**
- * Construct table schema from info stored in SSTable's Stats.db
- *
- * @param desc SSTable's descriptor
- * @return Restored TableMetadata
- * @throws IOException when Stats.db cannot be read
- */
-public static TableMetadata metadataFromSSTable(Descriptor desc) throws 
IOException
-{
-if (!desc.version.isCompatible())
-throw new IOException("Cannot process old and unsupported SSTable 
version.");
-
-EnumSet types = EnumSet.of(MetadataType.STATS, 
MetadataType.HEADER);
-Map sstableMetadata = 
desc.getMetadataSerializer().deserialize(desc, types);
-SerializationHeader.Component header = (SerializationHeader.Component) 
sstableMetadata.get(MetadataType.HEADER);
-IPartitioner partitioner = FBUtilities.newPartitioner(desc);
-
-TableMetadata.Builder builder = 

[jira] [Updated] (CASSANDRA-13737) Node start can fail if the base table of a materialized view is not found

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña updated CASSANDRA-13737:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Node start can fail if the base table of a materialized view is not found
> -
>
> Key: CASSANDRA-13737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13737
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Materialized Views
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Node start can fail if the base table of a materialized view is not found, 
> which is something that can happen under certain circumstances. There is a 
> dtest reproducing the problem:
> {code}
> cluster = self.cluster
> cluster.populate(3)
> cluster.start()
> node1, node2, node3 = self.cluster.nodelist()
> session = self.patient_cql_connection(node1, 
> consistency_level=ConsistencyLevel.QUORUM)
> create_ks(session, 'ks', 3)
> session.execute('CREATE TABLE users (username varchar PRIMARY KEY, state 
> varchar)')
> node3.stop(wait_other_notice=True)
> # create a materialized view only in nodes 1 and 2
> session.execute(('CREATE MATERIALIZED VIEW users_by_state AS '
>  'SELECT * FROM users WHERE state IS NOT NULL AND username IS 
> NOT NULL '
>  'PRIMARY KEY (state, username)'))
> node1.stop(wait_other_notice=True)
> node2.stop(wait_other_notice=True)
> # drop the base table only in node 3
> node3.start(wait_for_binary_proto=True)
> session = self.patient_cql_connection(node3, 
> consistency_level=ConsistencyLevel.QUORUM)
> session.execute('DROP TABLE ks.users')
> cluster.stop()
> cluster.start()  # Fails
> {code}
> This is the error during node start:
> {code}
> java.lang.IllegalArgumentException: Unknown CF 
> 958ebc30-76e4-11e7-869a-9d8367a71c76
>   at 
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:215) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:143) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:113) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.alterKeyspace(Schema.java:618) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.lambda$merge$18(Schema.java:591) 
> ~[main/:na]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
>  ~[na:1.8.0_131]
>   at java.util.HashMap$EntrySet.forEach(HashMap.java:1043) ~[na:1.8.0_131]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.forEach(Collections.java:1580)
>  ~[na:1.8.0_131]
>   at org.apache.cassandra.schema.Schema.merge(Schema.java:591) ~[main/:na]
>   at 
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) 
> ~[main/:na]
>   at 
> org.apache.cassandra.schema.MigrationTask$1.response(MigrationTask.java:89) 
> ~[main/:na]
>   at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
>  ~[main/:na]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) 
> ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_131]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [main/:na]
>   at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13737) Node start can fail if the base table of a materialized view is not found

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118103#comment-16118103
 ] 

ASF GitHub Bot commented on CASSANDRA-13737:


Github user adelapena closed the pull request at:

https://github.com/apache/cassandra-dtest/pull/4


> Node start can fail if the base table of a materialized view is not found
> -
>
> Key: CASSANDRA-13737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13737
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Materialized Views
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Node start can fail if the base table of a materialized view is not found, 
> which is something that can happen under certain circumstances. There is a 
> dtest reproducing the problem:
> {code}
> cluster = self.cluster
> cluster.populate(3)
> cluster.start()
> node1, node2, node3 = self.cluster.nodelist()
> session = self.patient_cql_connection(node1, 
> consistency_level=ConsistencyLevel.QUORUM)
> create_ks(session, 'ks', 3)
> session.execute('CREATE TABLE users (username varchar PRIMARY KEY, state 
> varchar)')
> node3.stop(wait_other_notice=True)
> # create a materialized view only in nodes 1 and 2
> session.execute(('CREATE MATERIALIZED VIEW users_by_state AS '
>  'SELECT * FROM users WHERE state IS NOT NULL AND username IS 
> NOT NULL '
>  'PRIMARY KEY (state, username)'))
> node1.stop(wait_other_notice=True)
> node2.stop(wait_other_notice=True)
> # drop the base table only in node 3
> node3.start(wait_for_binary_proto=True)
> session = self.patient_cql_connection(node3, 
> consistency_level=ConsistencyLevel.QUORUM)
> session.execute('DROP TABLE ks.users')
> cluster.stop()
> cluster.start()  # Fails
> {code}
> This is the error during node start:
> {code}
> java.lang.IllegalArgumentException: Unknown CF 
> 958ebc30-76e4-11e7-869a-9d8367a71c76
>   at 
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:215) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:143) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:113) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.alterKeyspace(Schema.java:618) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.lambda$merge$18(Schema.java:591) 
> ~[main/:na]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
>  ~[na:1.8.0_131]
>   at java.util.HashMap$EntrySet.forEach(HashMap.java:1043) ~[na:1.8.0_131]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.forEach(Collections.java:1580)
>  ~[na:1.8.0_131]
>   at org.apache.cassandra.schema.Schema.merge(Schema.java:591) ~[main/:na]
>   at 
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) 
> ~[main/:na]
>   at 
> org.apache.cassandra.schema.MigrationTask$1.response(MigrationTask.java:89) 
> ~[main/:na]
>   at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
>  ~[main/:na]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) 
> ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_131]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [main/:na]
>   at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13737) Node start can fail if the base table of a materialized view is not found

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118102#comment-16118102
 ] 

ASF GitHub Bot commented on CASSANDRA-13737:


Github user adelapena commented on the issue:

https://github.com/apache/cassandra-dtest/pull/4
  
Committed as 
[959208749d70e5808aec144e87b73e90d56a7f91](https://github.com/apache/cassandra-dtest/commit/959208749d70e5808aec144e87b73e90d56a7f91)


> Node start can fail if the base table of a materialized view is not found
> -
>
> Key: CASSANDRA-13737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13737
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Materialized Views
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Node start can fail if the base table of a materialized view is not found, 
> which is something that can happen under certain circumstances. There is a 
> dtest reproducing the problem:
> {code}
> cluster = self.cluster
> cluster.populate(3)
> cluster.start()
> node1, node2, node3 = self.cluster.nodelist()
> session = self.patient_cql_connection(node1, 
> consistency_level=ConsistencyLevel.QUORUM)
> create_ks(session, 'ks', 3)
> session.execute('CREATE TABLE users (username varchar PRIMARY KEY, state 
> varchar)')
> node3.stop(wait_other_notice=True)
> # create a materialized view only in nodes 1 and 2
> session.execute(('CREATE MATERIALIZED VIEW users_by_state AS '
>  'SELECT * FROM users WHERE state IS NOT NULL AND username IS 
> NOT NULL '
>  'PRIMARY KEY (state, username)'))
> node1.stop(wait_other_notice=True)
> node2.stop(wait_other_notice=True)
> # drop the base table only in node 3
> node3.start(wait_for_binary_proto=True)
> session = self.patient_cql_connection(node3, 
> consistency_level=ConsistencyLevel.QUORUM)
> session.execute('DROP TABLE ks.users')
> cluster.stop()
> cluster.start()  # Fails
> {code}
> This is the error during node start:
> {code}
> java.lang.IllegalArgumentException: Unknown CF 
> 958ebc30-76e4-11e7-869a-9d8367a71c76
>   at 
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:215) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:143) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:113) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.alterKeyspace(Schema.java:618) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.lambda$merge$18(Schema.java:591) 
> ~[main/:na]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
>  ~[na:1.8.0_131]
>   at java.util.HashMap$EntrySet.forEach(HashMap.java:1043) ~[na:1.8.0_131]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.forEach(Collections.java:1580)
>  ~[na:1.8.0_131]
>   at org.apache.cassandra.schema.Schema.merge(Schema.java:591) ~[main/:na]
>   at 
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) 
> ~[main/:na]
>   at 
> org.apache.cassandra.schema.MigrationTask$1.response(MigrationTask.java:89) 
> ~[main/:na]
>   at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
>  ~[main/:na]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) 
> ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_131]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [main/:na]
>   at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9988) Introduce leaf-only iterator

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118100#comment-16118100
 ] 

Benedict commented on CASSANDRA-9988:
-

bq. The original implementation is valid. I think that's another example shows 
the code is not that easy to understand.

Mea culpa, although I think that's more an indictment of my hubris and our 
mutual disinterest.  If fifteen distracted minutes in a JIRA comment window 
procrastinating from real work yields a minor bug, it's probably not *that* 
complicated, and an IDE, test suite and a couple of hours would probably more 
than conquer it.

bq. The graph is the search performance (throughput) for searching target on 
the different positions.

Except that this isn't the same, because the search domain is different with 
different starting positions.  This is also over-counting unrelated test setup 
costs.  One would expect that, with a leaf size of 32, iterating the whole 
contents would be _approx._ 4x faster with exp. search than binary search; the 
fact it is only 3x is probably down to those extra costs.  The worst inversion 
of performance would be around 2x (i.e. exp. search being 2x slower than 
b.search at its worst), with logarithmically spaced items - i.e. at position 
16, 8, 4, 2, 1.  However, in the slowest case for exp. search, we are doing 
much less work than the slowest for binary search.

Of course, with a limit to 32 items the exp. search benefits are much less 
apparent.  If we had a leaf size of 64 it would become much more stark.  With a 
leaf size of 16, it would also become completely irrelevant.



> Introduce leaf-only iterator
> 
>
> Key: CASSANDRA-9988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9988
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Benedict
>Assignee: Jay Zhuang
>Priority: Minor
>  Labels: patch
> Fix For: 4.0
>
> Attachments: 9988-3tests.png, 9988-data.png, 9988-result2.png, 
> 9988-result3.png, 9988-result.png, 9988-test-result3.png, 
> 9988-test-result-expsearch.xlsx, 9988-test-result-raw.png, 
> 9988-test-result.xlsx, 9988-trunk-new.txt, 9988-trunk-new-update.txt, 
> trunk-9988.txt
>
>
> In many cases we have small btrees, small enough to fit in a single leaf 
> page. In this case it _may_ be more efficient to specialise our iterator.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra-dtest git commit: Add test verifying that a schema propagation adding a view over a non existing table doesn't prevent a node from start

[adelapena]

Repository: cassandra-dtest
Updated Branches:
  refs/heads/master 7e3bcfd52 -> 959208749


Add test verifying that a schema propagation adding a view over a non existing 
table doesn't prevent a node from start

patch by Andres de la Peña; reviewed by Jake Luciani for CASSANDRA-13737


Project: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/commit/95920874
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/tree/95920874
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/diff/95920874

Branch: refs/heads/master
Commit: 959208749d70e5808aec144e87b73e90d56a7f91
Parents: 7e3bcfd
Author: Andrés de la Peña 
Authored: Tue Aug 8 10:01:15 2017 +0100
Committer: Andrés de la Peña 
Committed: Tue Aug 8 10:01:15 2017 +0100

--
 materialized_views_test.py | 35 +++
 1 file changed, 35 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/95920874/materialized_views_test.py
--
diff --git a/materialized_views_test.py b/materialized_views_test.py
index 306d719..77b20e6 100644
--- a/materialized_views_test.py
+++ b/materialized_views_test.py
@@ -1532,6 +1532,41 @@ class TestMaterializedViews(Tester):
 session.execute("DROP MATERIALIZED VIEW mv")
 session.execute("DROP TABLE test")
 
+def propagate_view_creation_over_non_existing_table(self):
+"""
+The internal addition of a view over a non existing table should be 
ignored
+@jira_ticket CASSANDRA-13737
+"""
+
+cluster = self.cluster
+cluster.populate(3)
+cluster.start()
+node1, node2, node3 = self.cluster.nodelist()
+session = self.patient_cql_connection(node1, 
consistency_level=ConsistencyLevel.QUORUM)
+create_ks(session, 'ks', 3)
+
+session.execute('CREATE TABLE users (username varchar PRIMARY KEY, 
state varchar)')
+
+# create a materialized view only in nodes 1 and 2
+node3.stop(wait_other_notice=True)
+session.execute(('CREATE MATERIALIZED VIEW users_by_state AS '
+ 'SELECT * FROM users WHERE state IS NOT NULL AND 
username IS NOT NULL '
+ 'PRIMARY KEY (state, username)'))
+
+# drop the base table only in node 3
+node1.stop(wait_other_notice=True)
+node2.stop(wait_other_notice=True)
+node3.start(wait_for_binary_proto=True)
+session = self.patient_cql_connection(node3, 
consistency_level=ConsistencyLevel.QUORUM)
+session.execute('DROP TABLE ks.users')
+
+# restart the cluster
+cluster.stop()
+cluster.start()
+
+# node3 should have received and ignored the creation of the MV over 
the dropped table
+self.assertTrue(node3.grep_log('Not adding view users_by_state because 
the base table'))
+
 
 # For read verification
 class MutationPresence(Enum):


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13067) Integer overflows with file system size reported by Amazon Elastic File System (EFS)

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-13067:

Status: Ready to Commit  (was: Patch Available)

> Integer overflows with file system size reported by Amazon Elastic File 
> System (EFS)
> 
>
> Key: CASSANDRA-13067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13067
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra in OpenShift running on Amazon EC2 instance 
> with EFS mounted for data
>Reporter: Michael Hanselmann
>Assignee: Benjamin Lerer
> Attachments: 0001-Handle-exabyte-sized-filesystems.patch
>
>
> When not explicitly configured Cassandra uses 
> [{{nio.FileStore.getTotalSpace}}|https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileStore.html]
>  to determine the total amount of available space in order to [calculate the 
> preferred commit log 
> size|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L553].
>  [Amazon EFS|https://aws.amazon.com/efs/] instances report a filesystem size 
> of 8 EiB when empty. [{{getTotalSpace}} causes an integer overflow 
> (JDK-8162520)|https://bugs.openjdk.java.net/browse/JDK-8162520] and returns a 
> negative number, resulting in a negative preferred size and causing the 
> checked integer to throw.
> Overriding {{commitlog_total_space_in_mb}} is not sufficient as 
> [{{DataDirectory.getAvailableSpace}}|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/db/Directories.java#L550]
>  makes use of {{nio.FileStore.getUsableSpace}}.
> [AMQ-6441] is a comparable issue in ActiveMQ.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13067) Integer overflows with file system size reported by Amazon Elastic File System (EFS)

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118013#comment-16118013
 ] 

Alex Petrov commented on CASSANDRA-13067:
-

Thank you for the patch and sorry it took me so long.

+1, LGTM. Tests look good as well.

> Integer overflows with file system size reported by Amazon Elastic File 
> System (EFS)
> 
>
> Key: CASSANDRA-13067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13067
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra in OpenShift running on Amazon EC2 instance 
> with EFS mounted for data
>Reporter: Michael Hanselmann
>Assignee: Benjamin Lerer
> Attachments: 0001-Handle-exabyte-sized-filesystems.patch
>
>
> When not explicitly configured Cassandra uses 
> [{{nio.FileStore.getTotalSpace}}|https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileStore.html]
>  to determine the total amount of available space in order to [calculate the 
> preferred commit log 
> size|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L553].
>  [Amazon EFS|https://aws.amazon.com/efs/] instances report a filesystem size 
> of 8 EiB when empty. [{{getTotalSpace}} causes an integer overflow 
> (JDK-8162520)|https://bugs.openjdk.java.net/browse/JDK-8162520] and returns a 
> negative number, resulting in a negative preferred size and causing the 
> checked integer to throw.
> Overriding {{commitlog_total_space_in_mb}} is not sufficient as 
> [{{DataDirectory.getAvailableSpace}}|https://github.com/apache/cassandra/blob/cassandra-3.9/src/java/org/apache/cassandra/db/Directories.java#L550]
>  makes use of {{nio.FileStore.getUsableSpace}}.
> [AMQ-6441] is a comparable issue in ActiveMQ.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13737) Node start can fail if the base table of a materialized view is not found

[Sergio Bossa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Bossa updated CASSANDRA-13737:
-
Reviewer: T Jake Luciani

> Node start can fail if the base table of a materialized view is not found
> -
>
> Key: CASSANDRA-13737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13737
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Materialized Views
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Node start can fail if the base table of a materialized view is not found, 
> which is something that can happen under certain circumstances. There is a 
> dtest reproducing the problem:
> {code}
> cluster = self.cluster
> cluster.populate(3)
> cluster.start()
> node1, node2, node3 = self.cluster.nodelist()
> session = self.patient_cql_connection(node1, 
> consistency_level=ConsistencyLevel.QUORUM)
> create_ks(session, 'ks', 3)
> session.execute('CREATE TABLE users (username varchar PRIMARY KEY, state 
> varchar)')
> node3.stop(wait_other_notice=True)
> # create a materialized view only in nodes 1 and 2
> session.execute(('CREATE MATERIALIZED VIEW users_by_state AS '
>  'SELECT * FROM users WHERE state IS NOT NULL AND username IS 
> NOT NULL '
>  'PRIMARY KEY (state, username)'))
> node1.stop(wait_other_notice=True)
> node2.stop(wait_other_notice=True)
> # drop the base table only in node 3
> node3.start(wait_for_binary_proto=True)
> session = self.patient_cql_connection(node3, 
> consistency_level=ConsistencyLevel.QUORUM)
> session.execute('DROP TABLE ks.users')
> cluster.stop()
> cluster.start()  # Fails
> {code}
> This is the error during node start:
> {code}
> java.lang.IllegalArgumentException: Unknown CF 
> 958ebc30-76e4-11e7-869a-9d8367a71c76
>   at 
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:215) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:143) 
> ~[main/:na]
>   at 
> org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:113) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.alterKeyspace(Schema.java:618) 
> ~[main/:na]
>   at org.apache.cassandra.schema.Schema.lambda$merge$18(Schema.java:591) 
> ~[main/:na]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
>  ~[na:1.8.0_131]
>   at java.util.HashMap$EntrySet.forEach(HashMap.java:1043) ~[na:1.8.0_131]
>   at 
> java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.forEach(Collections.java:1580)
>  ~[na:1.8.0_131]
>   at org.apache.cassandra.schema.Schema.merge(Schema.java:591) ~[main/:na]
>   at 
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) 
> ~[main/:na]
>   at 
> org.apache.cassandra.schema.MigrationTask$1.response(MigrationTask.java:89) 
> ~[main/:na]
>   at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
>  ~[main/:na]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) 
> ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_131]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_131]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
>   at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [main/:na]
>   at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13683) sstabledump should not use tool initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117921#comment-16117921
 ] 

ASF GitHub Bot commented on CASSANDRA-13683:


Github user asfgit closed the pull request at:

https://github.com/apache/cassandra/pull/130


> sstabledump should not use tool initialization
> --
>
> Key: CASSANDRA-13683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
> Fix For: 4.0
>
>
> The tool initialization is not necessary for the sstabledump utility and by 
> using it we possibly introduce a unnecessary requirement of being on the same 
> C* instance (since we may introduce changes that require it, UDT changes can 
> lead to dangerous assumptions) that has the schema. This also does things 
> like introduce possibly devastating issues from having the same 
> commitlog/sstable locations.
> A good example is updating the sstable activity table, and having that 
> flushed to commitlogs as an application outside the running C* service. This 
> may not be on same user (ie root) which than when a postmemtable flusher 
> after restart attempts to delete it will throw an exception and potentially 
> kill that thread pool. One that happens commitlogs will cease to get recycled 
> and will just burn disk space until we run out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13683) sstabledump should not use tool initialization

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13683:
---
   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Ready to Commit)

Thanks to both of you. Committed as {{9e3483f844d9db6fe2a6210550622fc2cd8aef72}}


> sstabledump should not use tool initialization
> --
>
> Key: CASSANDRA-13683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
> Fix For: 4.0
>
>
> The tool initialization is not necessary for the sstabledump utility and by 
> using it we possibly introduce a unnecessary requirement of being on the same 
> C* instance (since we may introduce changes that require it, UDT changes can 
> lead to dangerous assumptions) that has the schema. This also does things 
> like introduce possibly devastating issues from having the same 
> commitlog/sstable locations.
> A good example is updating the sstable activity table, and having that 
> flushed to commitlogs as an application outside the running C* service. This 
> may not be on same user (ie root) which than when a postmemtable flusher 
> after restart attempts to delete it will throw an exception and potentially 
> kill that thread pool. One that happens commitlogs will cease to get recycled 
> and will just burn disk space until we run out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13622) Better config validation/documentation

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117920#comment-16117920
 ] 

ZhaoYang commented on CASSANDRA-13622:
--

[~KurtG] could you review it?

> Better config validation/documentation
> --
>
> Key: CASSANDRA-13622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13622
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: Kurt Greaves
>Assignee: ZhaoYang
>Priority: Minor
>  Labels: lhf
> Fix For: 4.0
>
>
> There are a number of properties in the yaml that are "in_mb", however 
> resolve to bytes when calculated in {{DatabaseDescriptor.java}}, but are 
> stored in int's. This means that their maximum values are 2047, as any higher 
> when converted to bytes overflows the int.
> Where possible/reasonable we should convert these to be long's, and stored as 
> long's. If there is no reason for the value to ever be >2047 we should at 
> least document that as the max value, or better yet make it error if set 
> higher than that. Noting that although it's bad practice to increase a lot of 
> them to such high values, there may be cases where it is necessary and in 
> which case we should handle it appropriately rather than overflowing and 
> surprising the user. That is, causing it to break but not in the way the user 
> expected it to :)
> Following are functions that currently could be at risk of the above:
> {code:java|title=DatabaseDescriptor.java}
> getThriftFramedTransportSize()
> getMaxValueSize()
> getCompactionLargePartitionWarningThreshold()
> getCommitLogSegmentSize()
> getNativeTransportMaxFrameSize()
> # These are in KB so max value of 2096128
> getBatchSizeWarnThreshold()
> getColumnIndexSize()
> getColumnIndexCacheSize()
> getMaxMutationSize()
> {code}
> Note we may not actually need to fix all of these, and there may be more. 
> This was just from a rough scan over the code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Switch to client init for sstabledump

[jjirsa]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 6b302b600 -> 9e3483f84


Switch to client init for sstabledump

Closes #130

Patch by Chris Lohfink; Reviewed by Jeremiah Jordan for CASSANDRA-13683


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e3483f8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e3483f8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e3483f8

Branch: refs/heads/trunk
Commit: 9e3483f844d9db6fe2a6210550622fc2cd8aef72
Parents: 6b302b6
Author: Chris Lohfink 
Authored: Mon Jul 10 10:46:34 2017 -0500
Committer: Jeff Jirsa 
Committed: Mon Aug 7 23:23:30 2017 -0700

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/tools/SSTableExport.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e3483f8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b5a8dbb..69b8eaa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -103,6 +103,7 @@
  * Trivial format error in StorageProxy (CASSANDRA-13551)
  * Nodetool repair can hang forever if we lose the notification for the repair 
completing/failing (CASSANDRA-13480)
  * Anticompaction can cause noisy log messages (CASSANDRA-13684)
+ * Switch to client init for sstabledump (CASSANDRA-13683)
 
 
 3.11.1

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e3483f8/src/java/org/apache/cassandra/tools/SSTableExport.java
--
diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
b/src/java/org/apache/cassandra/tools/SSTableExport.java
index 913ee1f..5d22462 100644
--- a/src/java/org/apache/cassandra/tools/SSTableExport.java
+++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
@@ -63,7 +63,7 @@ public class SSTableExport
 
 static
 {
-DatabaseDescriptor.toolInitialization();
+DatabaseDescriptor.clientInitialization();
 
 Option optKey = new Option(KEY_OPTION, true, "Partition key");
 // Number of times -k  can be passed on the command line.


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13689) Update development docs with correct GH URL for new cassandra-dtest location

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13689:
---
   Resolution: Fixed
 Reviewer: Jeff Jirsa
Fix Version/s: 4.0
   Status: Resolved  (was: Ready to Commit)

Committed to trunk as 
[6b302b600f327cc8d1781776423d9548658e6591|https://github.com/apache/cassandra/commit/6b302b600f327cc8d1781776423d9548658e6591]
 , thanks!

(It'll take a while for the site to be rebuilt, but it's committed)



> Update development docs with correct GH URL for new cassandra-dtest location
> 
>
> Key: CASSANDRA-13689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13689
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Nate McCall
>Assignee: Roman Pogribnyi
>  Labels: docs-impacting, lhf
> Fix For: 4.0
>
> Attachments: 13689-3.11.txt
>
>
> Specifically the URL on this page:
> http://cassandra.apache.org/doc/latest/development/testing.html#dtests



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13689) Update development docs with correct GH URL for new cassandra-dtest location

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13689:
---
Status: Ready to Commit  (was: Patch Available)

> Update development docs with correct GH URL for new cassandra-dtest location
> 
>
> Key: CASSANDRA-13689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13689
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Nate McCall
>Assignee: Roman Pogribnyi
>  Labels: docs-impacting, lhf
> Attachments: 13689-3.11.txt
>
>
> Specifically the URL on this page:
> http://cassandra.apache.org/doc/latest/development/testing.html#dtests



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Update DTest GitHub URLs - Use ASF DTest location

[jjirsa]

Repository: cassandra
Updated Branches:
  refs/heads/trunk cb32807f6 -> 6b302b600


Update DTest GitHub URLs - Use ASF DTest location

Patch by Roman Pogribnyi; Reviewed by Jeff Jirsa for CASSANDRA-13689


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6b302b60
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6b302b60
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6b302b60

Branch: refs/heads/trunk
Commit: 6b302b600f327cc8d1781776423d9548658e6591
Parents: cb32807
Author: Roman Pogribnyi 
Authored: Sat Jul 22 22:06:01 2017 +0200
Committer: Jeff Jirsa 
Committed: Mon Aug 7 23:16:52 2017 -0700

--
 CONTRIBUTING.md| 2 +-
 doc/source/development/testing.rst | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6b302b60/CONTRIBUTING.md
--
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 25e15ee..779df1b 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -14,5 +14,5 @@ Use [Cassandra 
JIRA](https://issues.apache.org/jira/browse/CASSANDRA/) to create
 - Running Cassandra in IDEA 
[guide](https://wiki.apache.org/cassandra/RunningCassandraInIDEA)
 - Running Cassandra in Eclipse 
[guide](https://wiki.apache.org/cassandra/RunningCassandraInEclipse)
 - Cassandra Cluster Manager - [CCM](https://github.com/pcmanus/ccm) and a 
guide [blog 
post](http://www.datastax.com/dev/blog/ccm-a-development-tool-for-creating-local-cassandra-clusters)
-- Cassandra Distributed Tests aka 
[dtests](https://github.com/riptano/cassandra-dtest)
+- Cassandra Distributed Tests aka 
[dtests](https://github.com/apache/cassandra-dtest)
 - Cassandra Testing Guidelines - see TESTING.md

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6b302b60/doc/source/development/testing.rst
--
diff --git a/doc/source/development/testing.rst 
b/doc/source/development/testing.rst
index ca8b538..eb79932 100644
--- a/doc/source/development/testing.rst
+++ b/doc/source/development/testing.rst
@@ -60,11 +60,11 @@ Test that consume a significant amount of time during 
execution can be found in
 DTests
 ==
 
-One way of doing integration or system testing at larger scale is by using 
`dtest `_, which stands for 
“Cassandra Distributed Tests”. The idea is to automatically setup Cassandra 
clusters using various configurations and simulate certain use cases you want 
to test. This is done using Python scripts and ``ccmlib`` from the `ccm 
`_ project. Dtests will setup clusters using 
this library just as you do running ad-hoc ``ccm`` commands on your local 
machine. Afterwards dtests will use the `Python driver 
`_ to interact with 
the nodes, manipulate the file system, analyze logs or mess with individual 
nodes.
+One way of doing integration or system testing at larger scale is by using 
`dtest `_, which stands for 
“Cassandra Distributed Tests”. The idea is to automatically setup Cassandra 
clusters using various configurations and simulate certain use cases you want 
to test. This is done using Python scripts and ``ccmlib`` from the `ccm 
`_ project. Dtests will setup clusters using 
this library just as you do running ad-hoc ``ccm`` commands on your local 
machine. Afterwards dtests will use the `Python driver 
`_ to interact with 
the nodes, manipulate the file system, analyze logs or mess with individual 
nodes.
 
 Using dtests helps us to prevent regression bugs by continually executing 
tests on the `CI server `_ against new patches. 
Committers will be able to set up build branches there and your reviewer may 
use the CI environment to run tests for your patch. Read more on the motivation 
behind continuous integration `here 
`_.
 
-The best way to learn how to write dtests is probably by reading the 
introduction "`How to Write a Dtest 
`_" and by looking at 
existing, recently updated tests in the project. New tests must follow certain 
`style conventions 
`_ that 
are being checked before accepting contributions. In contrast to Cassandra, 
dtest issues and pull-requests are managed on github, therefor you should make 
sure