[jira] [Commented] (CASSANDRA-11464) C* doesn't respond to OPTIONS request containing low protocol number

[Sandeep Tamhankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222576#comment-15222576
 ] 

Sandeep Tamhankar commented on CASSANDRA-11464:
---

Looks good, but I noticed some debug statments (printing to stderr) that look 
like they were left in by accident.

> C* doesn't respond to OPTIONS request containing low protocol number
> 
>
> Key: CASSANDRA-11464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11464
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Sandeep Tamhankar
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Observed in Ruby and Python drivers: if you send an OPTIONS message to C* 3.4 
> (though I believe this goes back to 3.0) with a protocol version of 1, C* 
> doesn't send a response to the client. If you specify a high protocol version 
> (e.g. 5), C* does correctly respond with a protocol error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11464) C* doesn't respond to OPTIONS request containing low protocol number

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11464:

 Reviewer: Sandeep Tamhankar
Fix Version/s: 3.x
   3.0.x

The problem is that the v1/v2 protocol header is only 8 bytes, while the v3+ 
protocol header is 9 bytes.  When decoding a frame, we first wait for a 
complete header's worth of bytes.  When the actually header is only 8 bytes 
(instead of the expected 9), we never get enough bytes to trigger the protocol 
version check. I've adjusted this to check the protocol version as soon as we 
have the first byte.

It turns out that a dtest would be a little tricky to write due to the python 
driver now having automatic protocol version downgrading.  So, instead, I've 
added a new unit test that uses the shorter v1 protocol header.

Patch and pending CI test runs:

||branch||testall||dtest||
|[CASSANDRA-11464|https://github.com/thobbs/cassandra/tree/CASSANDRA-11464]|[testall|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11464-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11464-dtest]|
|[CASSANDRA-11464-trunk|https://github.com/thobbs/cassandra/tree/CASSANDRA-11464-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11464-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11464-trunk-dtest]|

[~stamhankar999] would you mind reviewing?

> C* doesn't respond to OPTIONS request containing low protocol number
> 
>
> Key: CASSANDRA-11464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11464
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Sandeep Tamhankar
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Observed in Ruby and Python drivers: if you send an OPTIONS message to C* 3.4 
> (though I believe this goes back to 3.0) with a protocol version of 1, C* 
> doesn't send a response to the client. If you specify a high protocol version 
> (e.g. 5), C* does correctly respond with a protocol error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11479) BatchlogManager unit tests failing on truncate race condition

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222438#comment-15222438
 ] 

Yuki Morishita edited comment on CASSANDRA-11479 at 4/1/16 10:23 PM:
-

Code-wise, It looks like compaction still can be running during 
{{runWithCompactionDisabled}}.

>From the log attached (Thank you!):

{noformat}
TRACE [MemtableFlushWriter:1] 2016-04-01 15:02:32,335 Scheduling a background 
task check for system.batches with SizeTieredCompactionStrategy
{noformat}

Here, compaction was submitted for flushed SSTable.

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Cancelling in-progress compactions for 
batches
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are []
TRACE [main] 2016-04-01 15:02:32,337 Compactions successfully cancelled
{noformat}

and cancelling compaction happened, but since compaction strategy (STCS) failed 
to grab compacting SSTables, compaction task was not yet created.

{code:java}
// interrupt in-progress compactions
CompactionManager.instance.interruptCompactionForCFs(selfWithAuxiliaryCfs, 
interruptValidation);
CompactionManager.instance.waitForCessation(selfWithAuxiliaryCfs);
{code}

did not happen since there was no {{CompactionTask}} and no SSTables were in 
compaction yet.

So,

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Discarding sstable data for truncated CF + 
indexes
{noformat}

truncate proceeded, but

{noformat}
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are 
[[BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-5-big-Data.db'),
 
BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-4-big-Data.db')]]
{noformat}

in CompactionExecutor, STCS was still trying to grab compaction candidate and 
it succeeded, putting those two into {{compacting}} SSTable.

EDIT: The first "Compaction buckets are: []" can be for repaired SSTables, and 
second one is for unrepaired.

Thus, inside truncate logic,

{code:java}
public ReplayPosition discardSSTables(long truncatedAt)
{
assert data.getCompacting().isEmpty() : data.getCompacting();

}
{code}
throws AssertionError claiming some SSTables are still compacting.

{noformat}
INFO  [CompactionExecutor:1] 2016-04-01 15:02:32,345 Compaction interrupted: 
Compaction@919a4bc5-7a33-3573-b03e-13fc3f68b465(system, batches, 0/240554)bytes
{noformat}

Eventually, the submitted {{CompactionTask}} was aborted since 
{{runWithCompactionDisabled}} marks compaction strategy to {{pause()}} at first.


was (Author: yukim):
Code-wise, It looks like compaction still can be running during 
{{runWithCompactionDisabled}}.

>From the log attached (Thank you!):

{noformat}
TRACE [MemtableFlushWriter:1] 2016-04-01 15:02:32,335 Scheduling a background 
task check for system.batches with SizeTieredCompactionStrategy
{noformat}

Here, compaction was submitted for flushed SSTable.

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Cancelling in-progress compactions for 
batches
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are []
TRACE [main] 2016-04-01 15:02:32,337 Compactions successfully cancelled
{noformat}

and cancelling compaction happened, but since compaction strategy (STCS) failed 
to grab compacting SSTables, compaction task was not yet created.

{code:java}
// interrupt in-progress compactions
CompactionManager.instance.interruptCompactionForCFs(selfWithAuxiliaryCfs, 
interruptValidation);
CompactionManager.instance.waitForCessation(selfWithAuxiliaryCfs);
{code}

did not happen since there was no {{CompactionTask}} and no SSTables were in 
compaction yet.

So,

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Discarding sstable data for truncated CF + 
indexes
{noformat}

truncate proceeded, but

{noformat}
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are 
[[BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-5-big-Data.db'),
 
BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-4-big-Data.db')]]
{noformat}

in CompactionExecutor, [STCS was still trying to grab compaction candidate 
eagerly|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L180]
 and it succeeded, putting those two into {{compacting}} SSTable.
Thus, inside truncate logic,

{code:java}
public ReplayPosition discardSSTables(long truncatedAt)
{
assert data.getCompacting().isEmpty() : data.getCompacting();

}
{code}
throws AssertionError claiming some SSTables are still compacting.

{noformat}
INFO  [CompactionExecutor:1] 2016-04-01 15:02:32,345 Compaction interrupted: 
Compaction@919a4bc5-7a33-3573-b03e-13fc3f68b465(system, batches, 0/240554)bytes
{noformat}

[jira] [Commented] (CASSANDRA-11479) BatchlogManager unit tests failing on truncate race condition

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222438#comment-15222438
 ] 

Yuki Morishita commented on CASSANDRA-11479:


Code-wise, It looks like compaction still can be running during 
{{runWithCompactionDisabled}}.

>From the log attached (Thank you!):

{noformat}
TRACE [MemtableFlushWriter:1] 2016-04-01 15:02:32,335 Scheduling a background 
task check for system.batches with SizeTieredCompactionStrategy
{noformat}

Here, compaction was submitted for flushed SSTable.

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Cancelling in-progress compactions for 
batches
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are []
TRACE [main] 2016-04-01 15:02:32,337 Compactions successfully cancelled
{noformat}

and cancelling compaction happened, but since compaction strategy (STCS) failed 
to grab compacting SSTables, compaction task was not yet created.

{code:java}
// interrupt in-progress compactions
CompactionManager.instance.interruptCompactionForCFs(selfWithAuxiliaryCfs, 
interruptValidation);
CompactionManager.instance.waitForCessation(selfWithAuxiliaryCfs);
{code}

did not happen since there was no {{CompactionTask}} and no SSTables were in 
compaction yet.

So,

{noformat}
TRACE [main] 2016-04-01 15:02:32,337 Discarding sstable data for truncated CF + 
indexes
{noformat}

truncate proceeded, but

{noformat}
TRACE [CompactionExecutor:1] 2016-04-01 15:02:32,337 Compaction buckets are 
[[BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-5-big-Data.db'),
 
BigTableReader(path='build/test/cassandra/data:0/system/batches-919a4bc57a333573b03e13fc3f68b465/ma-4-big-Data.db')]]
{noformat}

in CompactionExecutor, [STCS was still trying to grab compaction candidate 
eagerly|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L180]
 and it succeeded, putting those two into {{compacting}} SSTable.
Thus, inside truncate logic,

{code:java}
public ReplayPosition discardSSTables(long truncatedAt)
{
assert data.getCompacting().isEmpty() : data.getCompacting();

}
{code}
throws AssertionError claiming some SSTables are still compacting.

{noformat}
INFO  [CompactionExecutor:1] 2016-04-01 15:02:32,345 Compaction interrupted: 
Compaction@919a4bc5-7a33-3573-b03e-13fc3f68b465(system, batches, 0/240554)bytes
{noformat}

Eventually, the submitted {{CompactionTask}} was aborted since 
{{runWithCompactionDisabled}} marks compaction strategy to {{pause()}} at first.

> BatchlogManager unit tests failing on truncate race condition
> -
>
> Key: CASSANDRA-11479
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11479
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Joel Knighton
> Attachments: 
> TEST-org.apache.cassandra.batchlog.BatchlogManagerTest.log
>
>
> Example on CI 
> [here|http://cassci.datastax.com/job/trunk_testall/818/testReport/junit/org.apache.cassandra.batchlog/BatchlogManagerTest/testLegacyReplay_compression/].
>  This seems to have only started happening relatively recently (within the 
> last month or two).
> As far as I can tell, this is only showing up on BatchlogManagerTests purely 
> because it is an aggressive user of truncate. The assertion is hit in the 
> setUp method, so it can happen before any of the test methods. The assertion 
> occurs because a compaction is happening when truncate wants to discard 
> SSTables; trace level logs suggest that this compaction is submitted after 
> the pause on the CompactionStrategyManager.
> This should be reproducible by running BatchlogManagerTest in a loop - it 
> takes up to half an hour in my experience. A trace-level log from such a run 
> is attached - grep for my added log message "SSTABLES COMPACTING WHEN 
> DISCARDING" to find when the assert is hit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11389) Case sensitive in LIKE query althogh index created with false

[Pavel Yaskevich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Yaskevich resolved CASSANDRA-11389.
-
Resolution: Not A Bug
  Assignee: Pavel Yaskevich

> Case sensitive in LIKE query althogh index created with false
> -
>
> Key: CASSANDRA-11389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11389
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: Alon Levi
>Assignee: Pavel Yaskevich
>Priority: Minor
>  Labels: sasi
> Fix For: 3.x
>
>
> I created an index on user's first name as following: 
> CREATE CUSTOM INDEX ON users (first_name) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex'
> with options = {
> 'mode' : 'CONTAINS',
> 'case_sensitive' : 'false'
> };
> This is the data I have in my table
> user_id | first_name 
> | last_name
> ---+---+---
> daa312ae-ecdf-4eb4-b6e9-206e33e5ca24 |  Shlomo | Cohen
> ab38ce9d-2823-4e6a-994f-7783953baef1  |  Elad  |  Karakuli
> 5e8371a7-3ed9-479f-9e4b-e4a07c750b12 |  Alon  |  Levi
> ae85cdc0-5eb7-4f08-8e42-2abd89e327ed |  Gil | Elias
> Although i mentioned the option 'case_sensitive' : 'false'
> when I run this query : 
> select user_id, first_name from users where first_name LIKE '%shl%';
> The query returns no results.
> However, when I run this query :
> select user_id, first_name from users where first_name LIKE '%Shl%';
> The query returns the right results,
> and the strangest thing is when I run this query:
> select user_id, first_name from users where first_name LIKE 'shl%';
> suddenly the query is no more case sensitive and the results are fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11464) C* doesn't respond to OPTIONS request containing low protocol number

[Sandeep Tamhankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222372#comment-15222372
 ] 

Sandeep Tamhankar commented on CASSANDRA-11464:
---

Random theory I came up with about this: I wonder if C* doesn't parse the 
OPTIONS message properly because the stream-id is 8-bit (since I'm sending 
v1/v2), not 16-bit as it is in v3/v4. I could imagine that causing a nasty 
mis-parse that might confuse C* and keep it from responding.

> C* doesn't respond to OPTIONS request containing low protocol number
> 
>
> Key: CASSANDRA-11464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11464
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Sandeep Tamhankar
>Assignee: Tyler Hobbs
>
> Observed in Ruby and Python drivers: if you send an OPTIONS message to C* 3.4 
> (though I believe this goes back to 3.0) with a protocol version of 1, C* 
> doesn't send a response to the client. If you specify a high protocol version 
> (e.g. 5), C* does correctly respond with a protocol error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Tyler Hobbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222365#comment-15222365
 ] 

Tyler Hobbs commented on CASSANDRA-11349:
-

I think there are a couple of things wrong with the current patch.

First, the comparator needs to continue to compare the full cell name first, 
and only break ties on range tombstones with {{compare(t1.max, t2.max)}}.

Second, I believe we should remove the timestamp tie-breaking behavior from the 
comparator in general, and not just for validation compactions.  In other 
words, I think we're doing the comparison incorrectly for all compactions right 
now.

We want the comparison to return 0 whenever range tombstones have equal names 
and ranges, even if they have different timestamps.  This will result in 
{{LazilyCompactedRow.Reducer.reduce()}} being called in one round with each of 
the tombstones that only differ in timestamp.  The logic in 
{{LCR.Reducer.reduce()}} already handles the case of multiple range tombstones 
with different timestamps by picking the one with the highest timestamp, so 
these will correctly be reduced to a single RT.  It looks like the current 
codebase will keep both range tombstones during a compaction, which isn't 
necessarily harmful, but is suboptimal.  For repair purposes, though, this is 
incorrect as it produces a different digest.

To summarize: I think all we need to do is remove the timestamp tie-breaking 
logic from the existing comparator.

[~slebresne] should double-check my logic, though.

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11389) Case sensitive in LIKE query althogh index created with false

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222348#comment-15222348
 ] 

DOAN DuyHai commented on CASSANDRA-11389:
-

I faced more or less the same issue

I created an index with {{options =\{ 'mode' : 'PREFIX', 'case_sensitive' : 
'false' \}}} and of course case insensitivity was not taken into acount.

I dropped and re-created the index by adding an analyzer class (NonTokenizer) 
and it worked

> Case sensitive in LIKE query althogh index created with false
> -
>
> Key: CASSANDRA-11389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11389
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: Alon Levi
>Priority: Minor
>  Labels: sasi
> Fix For: 3.x
>
>
> I created an index on user's first name as following: 
> CREATE CUSTOM INDEX ON users (first_name) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex'
> with options = {
> 'mode' : 'CONTAINS',
> 'case_sensitive' : 'false'
> };
> This is the data I have in my table
> user_id | first_name 
> | last_name
> ---+---+---
> daa312ae-ecdf-4eb4-b6e9-206e33e5ca24 |  Shlomo | Cohen
> ab38ce9d-2823-4e6a-994f-7783953baef1  |  Elad  |  Karakuli
> 5e8371a7-3ed9-479f-9e4b-e4a07c750b12 |  Alon  |  Levi
> ae85cdc0-5eb7-4f08-8e42-2abd89e327ed |  Gil | Elias
> Although i mentioned the option 'case_sensitive' : 'false'
> when I run this query : 
> select user_id, first_name from users where first_name LIKE '%shl%';
> The query returns no results.
> However, when I run this query :
> select user_id, first_name from users where first_name LIKE '%Shl%';
> The query returns the right results,
> and the strangest thing is when I run this query:
> select user_id, first_name from users where first_name LIKE 'shl%';
> suddenly the query is no more case sensitive and the results are fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11183) Enable SASI index for static columns

[DOAN DuyHai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DOAN DuyHai reassigned CASSANDRA-11183:
---

Assignee: DOAN DuyHai

> Enable SASI index for static columns
> 
>
> Key: CASSANDRA-11183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11183
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: DOAN DuyHai
>Assignee: DOAN DuyHai
>Priority: Minor
>
> This is a follow up ticket for post Cassandra 3.4 SASI integration.
> Since [CASSANDRA-8103] it is possible to index static columns, which is 
> *extremely useful* for some scenarios (find all sensors whose characteristics 
> are saved in static columns)
> /cc [~xedin] [~rustyrazorblade] [~jkrupan]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Richard Low (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222330#comment-15222330
 ] 

Richard Low commented on CASSANDRA-11349:
-

I'm also not sure how this is meant to fix it. Special casing validation 
compaction may fix repairs but you'd still get the digest mismatches on reads.

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11479) BatchlogManager unit tests failing on truncate race condition

[Joel Knighton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-11479:
--
Attachment: TEST-org.apache.cassandra.batchlog.BatchlogManagerTest.log

> BatchlogManager unit tests failing on truncate race condition
> -
>
> Key: CASSANDRA-11479
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11479
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Joel Knighton
> Attachments: 
> TEST-org.apache.cassandra.batchlog.BatchlogManagerTest.log
>
>
> Example on CI 
> [here|http://cassci.datastax.com/job/trunk_testall/818/testReport/junit/org.apache.cassandra.batchlog/BatchlogManagerTest/testLegacyReplay_compression/].
>  This seems to have only started happening relatively recently (within the 
> last month or two).
> As far as I can tell, this is only showing up on BatchlogManagerTests purely 
> because it is an aggressive user of truncate. The assertion is hit in the 
> setUp method, so it can happen before any of the test methods. The assertion 
> occurs because a compaction is happening when truncate wants to discard 
> SSTables; trace level logs suggest that this compaction is submitted after 
> the pause on the CompactionStrategyManager.
> This should be reproducible by running BatchlogManagerTest in a loop - it 
> takes up to half an hour in my experience. A trace-level log from such a run 
> is attached - grep for my added log message "SSTABLES COMPACTING WHEN 
> DISCARDING" to find when the assert is hit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11479) BatchlogManager unit tests failing on truncate race condition

[Joel Knighton (JIRA)]

Joel Knighton created CASSANDRA-11479:
-

 Summary: BatchlogManager unit tests failing on truncate race 
condition
 Key: CASSANDRA-11479
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11479
 Project: Cassandra
  Issue Type: Bug
Reporter: Joel Knighton


Example on CI 
[here|http://cassci.datastax.com/job/trunk_testall/818/testReport/junit/org.apache.cassandra.batchlog/BatchlogManagerTest/testLegacyReplay_compression/].
 This seems to have only started happening relatively recently (within the last 
month or two).

As far as I can tell, this is only showing up on BatchlogManagerTests purely 
because it is an aggressive user of truncate. The assertion is hit in the setUp 
method, so it can happen before any of the test methods. The assertion occurs 
because a compaction is happening when truncate wants to discard SSTables; 
trace level logs suggest that this compaction is submitted after the pause on 
the CompactionStrategyManager.

This should be reproducible by running BatchlogManagerTest in a loop - it takes 
up to half an hour in my experience. A trace-level log from such a run is 
attached - grep for my added log message "SSTABLES COMPACTING WHEN DISCARDING" 
to find when the assert is hit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11417) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc

[Jim Witschey (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222311#comment-15222311
 ] 

Jim Witschey commented on CASSANDRA-11417:
--

Closing - we'll continue keeping an eye on the test, and if it fails again 
we'll open a new ticket..

> dtest failure in 
> replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc
> --
>
> Key: CASSANDRA-11417
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11417
> Project: Cassandra
>  Issue Type: Test
>Reporter: Philip Thompson
>Assignee: Jim Witschey
>  Labels: dtest
>
> Error is 
> {code}
> Unknown table 'rf_test' in keyspace 'testing'
> {code}
> Just seems like a schema disagreement problem. Presumably we just need to 
> have the driver block until schema agreement.
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/90/testReport/replication_test/SnitchConfigurationUpdateTest/test_rf_expand_gossiping_property_file_snitch_multi_dc
> Failed on CassCI build trunk_offheap_dtest #90



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11417) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc

[Jim Witschey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Witschey resolved CASSANDRA-11417.
--
Resolution: Fixed

> dtest failure in 
> replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc
> --
>
> Key: CASSANDRA-11417
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11417
> Project: Cassandra
>  Issue Type: Test
>Reporter: Philip Thompson
>Assignee: Jim Witschey
>  Labels: dtest
>
> Error is 
> {code}
> Unknown table 'rf_test' in keyspace 'testing'
> {code}
> Just seems like a schema disagreement problem. Presumably we just need to 
> have the driver block until schema agreement.
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/90/testReport/replication_test/SnitchConfigurationUpdateTest/test_rf_expand_gossiping_property_file_snitch_multi_dc
> Failed on CassCI build trunk_offheap_dtest #90



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222304#comment-15222304
 ] 

Michael Kjellman commented on CASSANDRA-11349:
--

And just for my sanity and for discussion in the Jira, here is the current 
handling in the comparator

{code}
if (c1 instanceof RangeTombstone)
{
if (c2 instanceof RangeTombstone)
{
RangeTombstone t1 = (RangeTombstone)c1;
RangeTombstone t2 = (RangeTombstone)c2;
int comp2 = AbstractCellNameType.this.compare(t1.max, t2.max);
return comp2 == 0 ? t1.data.compareTo(t2.data) : comp2;
}
else
{
return -1;
}
}
else
{
return c2 instanceof RangeTombstone ? 1 : 0;
}
}
{code}

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1597#comment-1597
 ] 

Michael Kjellman commented on CASSANDRA-11349:
--

[~spo...@gmail.com] [~slebresne] [~frousseau] I'm confused here. Why should 
repair be special cased over normal compaction in this case? If the times are 
different then you *do* still need to resolve it as you need to take the 
greater time.

It seems to me the crux of the current patch is to "fix" this by special casing 
the comparator to just compare just the max value of the interval during repair 
validation:

{code}
// only compare interval, but not deletion time
+  return AbstractCellNameType.this.compare(((RangeTombstone)c1).max, 
((RangeTombstone)c2).max);
{code}

I just did my best to merge and compare the code between 2.0 and 2.1 and I'm 
still trying to parse how this code is different in 2.0 vs. 2.1... We've been 
unable to reproduce this in 2.0 so far, but the bits of the code being touched 
here don't seem to be different so I'm trying to understand why 2.1 would hit 
this and not 2.0.

Could you please explain a bit more why we we can ignore the timestamp?


> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7017) allow per-partition LIMIT clause in cql

[Dan Burkert (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dan Burkert updated CASSANDRA-7017:
---
Assignee: (was: Dan Burkert)

> allow per-partition LIMIT clause in cql
> ---
>
> Key: CASSANDRA-7017
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7017
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jonathan Halliday
>  Labels: cql
> Fix For: 3.x
>
> Attachments: 0001-CASSANDRA-7017.patch
>
>
> somewhat related to static columns (#6561) and slicing (#4851), it is 
> desirable to apply a LIMIT on a per-partition rather than per-query basis, 
> such as to retrieve the top (most recent, etc) N clustered values for each 
> partition key, e.g.
> -- for each league, keep a ranked list of users
> create table scores (league text, score int, player text, primary key(league, 
> score, player) );
> -- get the top 3 teams in each league:
> select * from scores staticlimit 3;
> this currently requires issuing one query per partition key, which is tedious 
> if all the key partition key values are known and impossible if they aren't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs resolved CASSANDRA-11320.
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 3.x)
   3.5
   3.0.5

bq. I believe they should be fine as hidden options?

Yes, that seems fine to me.  I forgot that the dtest was using them.

bq. Is there a better way?

I'm not sure, I didn't spend a lot of time digging into it.  That seems fine to 
me for now, though.

bq. We could back-port it to 2.2 as well.

Since this is a fairly large change, let's keep it in 3.x for now.  If we see 
this problem in the wild in 2.2, we can look at backporting it.

bq. once the code review is completed, I will take care of merging to other 
branches and running CI.

In order to get this before the 3.5 code freeze, I went ahead and merged this 
myself and ran the relevant dtests locally.  Aside from one [minor dtest 
tweak|https://github.com/riptano/cassandra-dtest/commit/0b0047617c8a38562d3714e1e6b9d744396b255f]
 everything looks good.

So, +1, committed as {{98086b65d0bc76631a3aeb50cddd8c9a82bc05b9}} to 3.0 and 
merged upwards.  Thanks!

> Improve backoff policy for cqlsh COPY FROM
> --
>
> Key: CASSANDRA-11320
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11320
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
>  Labels: doc-impacting
> Fix For: 3.0.5, 3.5
>
>
> Currently we have an exponential back-off policy in COPY FROM that kicks in 
> when timeouts are received. However there are two limitations:
> * it does not cover new requests and therefore we may not back-off 
> sufficiently to give time to an overloaded server to recover
> * the pause is performed in the receiving thread and therefore we may not 
> process server messages quickly enough
> There is a static throttling mechanism in rows per second from feeder to 
> worker processes (the INGESTRATE) but the feeder has no idea of the load of 
> each worker process. However it's easy to keep track of how many chunks a 
> worker process has yet to read by introducing a bounded semaphore.
> The idea is to move the back-off pauses to the worker processes main thread 
> so as to include all messages, new and retries, not just the retries that 
> timed out. The worker process will not read new chunks during the back-off 
> pauses, and the feeder process can then look at the number of pending chunks 
> before sending new chunks to a worker process.
> [~aholmber], [~aweisberg] what do you think?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11312) DatabaseDescriptor#applyConfig should log stacktrace in case of Eception during seed provider creation

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-11312:

Assignee: Andrzej Ludwikowski

> DatabaseDescriptor#applyConfig should log stacktrace in case of Eception 
> during seed provider creation
> --
>
> Key: CASSANDRA-11312
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11312
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Andrzej Ludwikowski
>Assignee: Andrzej Ludwikowski
>Priority: Minor
> Fix For: 2.2.6, 3.0.5, 3.5, 3.6
>
> Attachments: 11312-2.2.txt
>
>
> The comment says that: "See log for stacktrace.", but with the the flag 
> "false" stacktrace is not logged. Logging stacktrace will save some time when 
> someone (like me) mess up with the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11461) Failed incremental repairs never cleared from pending list

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222142#comment-15222142
 ] 

Marcus Eriksson commented on CASSANDRA-11461:
-

bq. prevent full incremental repairs from ever running on more than one node
With vnodes, yes. Or, I guess with large clusters and small number of tokens 
per node it is possible to find nodes that don't share any ranges

bq. never allow subrange incremental repairs
yep, it does not really make sense since we don't anticompact

> Failed incremental repairs never cleared from pending list
> --
>
> Key: CASSANDRA-11461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11461
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Adam Hattrell
>
> Set up a test cluster with 2 DC's, heavy use of LCS (not sure if that's 
> relevant).
> Kick off cassandra-stress against it.
> Kick of an automated incremental repair cycle.  
> After a bit a node starts flapping which causes a few repairs to fail.  This 
> is never cleared out of pending repairs - given the keyspace is replicated to 
> all nodes it means they all have pending repairs that will never complete.  
> Repairs  are basically blocked at this point.
> Given we're using Incremental repairs you're now spammed with:
> "Cannot start multiple repair sessions over the same sstables"
> Cluster and logs are still available for review - message me for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11461) Failed incremental repairs never cleared from pending list

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson resolved CASSANDRA-11461.
-
   Resolution: Not A Problem
Reproduced In: 2.1.13, 2.1.12  (was: 2.1.12, 2.1.13)

closing as not a problem as it seems this is an OpsCenter issue

> Failed incremental repairs never cleared from pending list
> --
>
> Key: CASSANDRA-11461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11461
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Adam Hattrell
>
> Set up a test cluster with 2 DC's, heavy use of LCS (not sure if that's 
> relevant).
> Kick off cassandra-stress against it.
> Kick of an automated incremental repair cycle.  
> After a bit a node starts flapping which causes a few repairs to fail.  This 
> is never cleared out of pending repairs - given the keyspace is replicated to 
> all nodes it means they all have pending repairs that will never complete.  
> Repairs  are basically blocked at this point.
> Given we're using Incremental repairs you're now spammed with:
> "Cannot start multiple repair sessions over the same sstables"
> Cluster and logs are still available for review - message me for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11312) DatabaseDescriptor#applyConfig should log stacktrace in case of Eception during seed provider creation

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-11312.
-
   Resolution: Fixed
 Reviewer: Jason Brown
Fix Version/s: 3.6
   3.5
   3.0.5
   2.2.6

committed as 0ac2072bb2cc6d8e069e07f5cbcdf2e83cdc5b5c

> DatabaseDescriptor#applyConfig should log stacktrace in case of Eception 
> during seed provider creation
> --
>
> Key: CASSANDRA-11312
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11312
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Andrzej Ludwikowski
>Priority: Minor
> Fix For: 2.2.6, 3.0.5, 3.5, 3.6
>
> Attachments: 11312-2.2.txt
>
>
> The comment says that: "See log for stacktrace.", but with the the flag 
> "false" stacktrace is not logged. Logging stacktrace will save some time when 
> someone (like me) mess up with the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/9] cassandra git commit: DatabaseDescriptor should log stacktrace in case of Eception during seed provider creation

[jasobrown]

DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation

patch by Andrzej Ludwikowski; reviewed by jasobrown for CASSANDRA-11312


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ac2072b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ac2072b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ac2072b

Branch: refs/heads/cassandra-3.5
Commit: 0ac2072bb2cc6d8e069e07f5cbcdf2e83cdc5b5c
Parents: c662259
Author: Andrzej Ludwikowski 
Authored: Mon Mar 7 19:27:54 2016 +0100
Committer: Jason Brown 
Committed: Fri Apr 1 11:16:35 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c8a4f21..28903c5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 5ad62c3..46e832d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -669,7 +669,7 @@ public class DatabaseDescriptor
 // there are about 5 checked exceptions that could be thrown here.
 catch (Exception e)
 {
-throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", false);
+throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", true);
 }
 if (seedProvider.getSeeds().size() == 0)
 throw new ConfigurationException("The seed provider lists no 
seeds.", false);

[6/9] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d3f6f943
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d3f6f943
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d3f6f943

Branch: refs/heads/trunk
Commit: d3f6f9438f21019a0c499047489048d3d0aee7b8
Parents: 1a1f836 0ac2072
Author: Jason Brown 
Authored: Fri Apr 1 11:30:50 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:31:23 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/CHANGES.txt
--
diff --cc CHANGES.txt
index 0ce75a8,28903c5..8305852
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,23 -1,10 +1,24 @@@
 -2.2.6
 +3.0.5
 + * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
   * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[5/9] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d3f6f943
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d3f6f943
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d3f6f943

Branch: refs/heads/cassandra-3.5
Commit: d3f6f9438f21019a0c499047489048d3d0aee7b8
Parents: 1a1f836 0ac2072
Author: Jason Brown 
Authored: Fri Apr 1 11:30:50 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:31:23 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/CHANGES.txt
--
diff --cc CHANGES.txt
index 0ce75a8,28903c5..8305852
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,23 -1,10 +1,24 @@@
 -2.2.6
 +3.0.5
 + * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
   * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[8/9] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/18f8c2fe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/18f8c2fe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/18f8c2fe

Branch: refs/heads/trunk
Commit: 18f8c2feab449d7e202f877f6fdd6c3cb9d6bcaa
Parents: 020a80b d3f6f94
Author: Jason Brown 
Authored: Fri Apr 1 11:32:12 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:32:43 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/18f8c2fe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/18f8c2fe/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[7/9] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/18f8c2fe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/18f8c2fe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/18f8c2fe

Branch: refs/heads/cassandra-3.5
Commit: 18f8c2feab449d7e202f877f6fdd6c3cb9d6bcaa
Parents: 020a80b d3f6f94
Author: Jason Brown 
Authored: Fri Apr 1 11:32:12 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:32:43 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/18f8c2fe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/18f8c2fe/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[9/9] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

[jasobrown]

Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/204cf347
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/204cf347
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/204cf347

Branch: refs/heads/trunk
Commit: 204cf3478e76ce9c41dbb1adabc52abc7c09133a
Parents: ee9fc22 18f8c2f
Author: Jason Brown 
Authored: Fri Apr 1 11:33:11 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:33:35 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/204cf347/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/204cf347/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[4/9] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d3f6f943
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d3f6f943
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d3f6f943

Branch: refs/heads/cassandra-3.0
Commit: d3f6f9438f21019a0c499047489048d3d0aee7b8
Parents: 1a1f836 0ac2072
Author: Jason Brown 
Authored: Fri Apr 1 11:30:50 2016 -0700
Committer: Jason Brown 
Committed: Fri Apr 1 11:31:23 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/CHANGES.txt
--
diff --cc CHANGES.txt
index 0ce75a8,28903c5..8305852
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,23 -1,10 +1,24 @@@
 -2.2.6
 +3.0.5
 + * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
   * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3f6f943/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

[1/9] cassandra git commit: DatabaseDescriptor should log stacktrace in case of Eception during seed provider creation

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 1a1f836f2 -> d3f6f9438
  refs/heads/cassandra-3.5 020a80b13 -> 18f8c2fea
  refs/heads/trunk ee9fc22a6 -> 204cf3478


DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation

patch by Andrzej Ludwikowski; reviewed by jasobrown for CASSANDRA-11312


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ac2072b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ac2072b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ac2072b

Branch: refs/heads/cassandra-3.0
Commit: 0ac2072bb2cc6d8e069e07f5cbcdf2e83cdc5b5c
Parents: c662259
Author: Andrzej Ludwikowski 
Authored: Mon Mar 7 19:27:54 2016 +0100
Committer: Jason Brown 
Committed: Fri Apr 1 11:16:35 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c8a4f21..28903c5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 5ad62c3..46e832d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -669,7 +669,7 @@ public class DatabaseDescriptor
 // there are about 5 checked exceptions that could be thrown here.
 catch (Exception e)
 {
-throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", false);
+throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", true);
 }
 if (seedProvider.getSeeds().size() == 0)
 throw new ConfigurationException("The seed provider lists no 
seeds.", false);

[3/9] cassandra git commit: DatabaseDescriptor should log stacktrace in case of Eception during seed provider creation

[jasobrown]

DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation

patch by Andrzej Ludwikowski; reviewed by jasobrown for CASSANDRA-11312


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ac2072b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ac2072b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ac2072b

Branch: refs/heads/trunk
Commit: 0ac2072bb2cc6d8e069e07f5cbcdf2e83cdc5b5c
Parents: c662259
Author: Andrzej Ludwikowski 
Authored: Mon Mar 7 19:27:54 2016 +0100
Committer: Jason Brown 
Committed: Fri Apr 1 11:16:35 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c8a4f21..28903c5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 5ad62c3..46e832d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -669,7 +669,7 @@ public class DatabaseDescriptor
 // there are about 5 checked exceptions that could be thrown here.
 catch (Exception e)
 {
-throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", false);
+throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", true);
 }
 if (seedProvider.getSeeds().size() == 0)
 throw new ConfigurationException("The seed provider lists no 
seeds.", false);

cassandra git commit: DatabaseDescriptor should log stacktrace in case of Eception during seed provider creation

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 c662259fe -> 0ac2072bb


DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation

patch by Andrzej Ludwikowski; reviewed by jasobrown for CASSANDRA-11312


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ac2072b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ac2072b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ac2072b

Branch: refs/heads/cassandra-2.2
Commit: 0ac2072bb2cc6d8e069e07f5cbcdf2e83cdc5b5c
Parents: c662259
Author: Andrzej Ludwikowski 
Authored: Mon Mar 7 19:27:54 2016 +0100
Committer: Jason Brown 
Committed: Fri Apr 1 11:16:35 2016 -0700

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c8a4f21..28903c5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * DatabaseDescriptor should log stacktrace in case of Eception during seed 
provider creation (CASSANDRA-11312)
  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ac2072b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 5ad62c3..46e832d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -669,7 +669,7 @@ public class DatabaseDescriptor
 // there are about 5 checked exceptions that could be thrown here.
 catch (Exception e)
 {
-throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", false);
+throw new ConfigurationException(e.getMessage() + "\nFatal 
configuration error; unable to start server.  See log for stacktrace.", true);
 }
 if (seedProvider.getSeeds().size() == 0)
 throw new ConfigurationException("The seed provider lists no 
seeds.", false);

[jira] [Updated] (CASSANDRA-10943) NullPointer during LegacySchemaMigrator

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-10943:
--
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 3.x)
   3.5
   3.0.5
   Status: Resolved  (was: Patch Available)

Committed as 
[1a1f836f20ba4bf1c263d4fd3c32338e55abbcbc|https://github.com/apache/cassandra/commit/1a1f836f20ba4bf1c263d4fd3c32338e55abbcbc]
 to 3.0 and merged with 3.5 and trunk, thanks.

> NullPointer during LegacySchemaMigrator
> ---
>
> Key: CASSANDRA-10943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10943
> Project: Cassandra
>  Issue Type: Bug
> Environment: Debian Jessie
> java version "1.8.0_66"
> Cassandra 2.2.4 -> 3.1.1 migration
>Reporter: Stephan Wienczny
>Assignee: Aleksey Yeschenko
>Priority: Minor
> Fix For: 3.0.5, 3.5
>
> Attachments: 10943.txt
>
>
> I'm trying to upgrade my Cassandra cluster  from 2.2.4 to 3.1.1.
> I used
> {quote}
> nodetool upgradesstables
> nodetool drain
> {quote}
> before upgrading.  
> The result is this:
> {quote}
> INFO  [main] 2015-12-26 22:41:44,114 SystemKeyspace.java:1284 - Detected 
> version upgrade from 2.2.4 to 3.1.1, snapshotting system keyspace
> WARN  [main] 2015-12-26 22:41:44,318 CompressionParams.java:382 - The 
> sstable_compression option has been deprecated. You should use class instead
> ERROR [main] 2015-12-26 22:41:44,427 CassandraDaemon.java:690 - Exception 
> encountered during startup
> java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:33)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:24)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:114) 
> ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:272)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:264)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:250)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:221)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$218(LegacySchemaMigrator.java:214)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:214)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:163)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$215(LegacySchemaMigrator.java:154)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:154)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:223) 
> [apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:549)
>  [apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:677) 
> [apache-cassandra-3.1.1.jar:3.1.1]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[6/6] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

[aleksey]

Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee9fc22a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee9fc22a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee9fc22a

Branch: refs/heads/trunk
Commit: ee9fc22a6467af61a2dd5b91c1e8839b63f5e907
Parents: fc37e61 020a80b
Author: Aleksey Yeschenko 
Authored: Fri Apr 1 19:21:11 2016 +0100
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:21:30 2016 +0100

--
 CHANGES.txt |  2 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 64 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee9fc22a/CHANGES.txt
--
diff --cc CHANGES.txt
index d79111f,61265b1..a0ece81
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,35 -1,3 +1,36 @@@
 +3.6
 + * Support LIKE operator in prepared statements (CASSANDRA-11456)
 + * Add a command to see if a Materialized View has finished building 
(CASSANDRA-9967)
 + * Log endpoint and port associated with streaming operation (CASSANDRA-8777)
 + * Print sensible units for all log messages (CASSANDRA-9692)
 + * Upgrade Netty to version 4.0.34 (CASSANDRA-11096)
 + * Break the CQL grammar into separate Parser and Lexer (CASSANDRA-11372)
 + * Compress only inter-dc traffic by default (CASSANDRA-)
 + * Add metrics to track write amplification (CASSANDRA-11420)
 + * cassandra-stress: cannot handle "value-less" tables (CASSANDRA-7739)
 + * Add/drop multiple columns in one ALTER TABLE statement (CASSANDRA-10411)
 + * Add require_endpoint_verification opt for internode encryption 
(CASSANDRA-9220)
 + * Add auto import java.util for UDF code block (CASSANDRA-11392)
 + * Add --hex-format option to nodetool getsstables (CASSANDRA-11337)
 + * sstablemetadata should print sstable min/max token (CASSANDRA-7159)
 + * Do not wrap CassandraException in TriggerExecutor (CASSANDRA-9421)
 + * COPY TO should have higher double precision (CASSANDRA-11255)
 + * Stress should exit with non-zero status after failure (CASSANDRA-10340)
 + * Add client to cqlsh SHOW_SESSION (CASSANDRA-8958)
 + * Fix nodetool tablestats keyspace level metrics (CASSANDRA-11226)
 + * Store repair options in parent_repair_history (CASSANDRA-11244)
 + * Print current leveling in sstableofflinerelevel (CASSANDRA-9588)
 + * Change repair message for keyspaces with RF 1 (CASSANDRA-11203)
 + * Remove hard-coded SSL cipher suites and protocols (CASSANDRA-10508)
 + * Improve concurrency in CompactionStrategyManager (CASSANDRA-10099)
 + * (cqlsh) interpret CQL type for formatting blobs (CASSANDRA-11274)
 + * Refuse to start and print txn log information in case of disk
 +   corruption (CASSANDRA-10112)
 + * Resolve some eclipse-warnings (CASSANDRA-11086)
 + * (cqlsh) Show static columns in a different color (CASSANDRA-11059)
 + * Allow to remove TTLs on table with default_time_to_live (CASSANDRA-11207)
 +
++
  3.5
   * Avoid index segment stitching in RAM which lead to OOM on big SSTable 
files (CASSANDRA-11383)
   * Fix clustering and row filters for LIKE queries on clustering columns 
(CASSANDRA-11397)

[2/6] cassandra git commit: Fix rare NPE on schema upgrade from 2.x to 3.x

[aleksey]

Fix rare NPE on schema upgrade from 2.x to 3.x

patch by Aleksey Yeschenko; reviewed by Sylvain Lebresne for
CASSANDRA-10943


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a1f836f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a1f836f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a1f836f

Branch: refs/heads/cassandra-3.5
Commit: 1a1f836f20ba4bf1c263d4fd3c32338e55abbcbc
Parents: 1513139
Author: Aleksey Yeschenko 
Authored: Wed Mar 23 21:21:29 2016 +
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:19:51 2016 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8d4834..0ce75a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
  * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--
diff --git a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java 
b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
index afa0f38..7411b93 100644
--- a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
+++ b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
@@ -284,7 +284,9 @@ public final class LegacySchemaMigrator
 AbstractType subComparator = tableRow.has("subcomparator") ? 
TypeParser.parse(tableRow.getString("subcomparator")) : null;
 
 boolean isSuper = 
"super".equals(tableRow.getString("type").toLowerCase());
-boolean isDense = tableRow.getBoolean("is_dense");
+boolean isDense = tableRow.has("is_dense")
+? tableRow.getBoolean("is_dense")
+: calculateIsDense(rawComparator, columnRows);
 boolean isCompound = rawComparator instanceof CompositeType;
 
 // We don't really use the default validator but as we have it for 
backward compatibility, we use it to know if it's a counter table
@@ -364,6 +366,65 @@ public final class LegacySchemaMigrator
   .triggers(createTriggersFromTriggerRows(triggerRows));
 }
 
+/*
+ * We call dense a CF for which each component of the comparator is a 
clustering column, i.e. no
+ * component is used to store a regular column names. In other words, 
non-composite static "thrift"
+ * and CQL3 CF are *not* dense.
+ * We save whether the table is dense or not during table creation through 
CQL, but we don't have this
+ * information for table just created through thrift, nor for table prior 
to CASSANDRA-7744, so this
+ * method does its best to infer whether the table is dense or not based 
on other elements.
+ */
+public static boolean calculateIsDense(AbstractType comparator, 
UntypedResultSet columnRows)
+{
+/*
+ * As said above, this method is only here because we need to deal 
with thrift upgrades.
+ * Once a CF has been "upgraded", i.e. we've rebuilt and save its CQL3 
metadata at least once,
+ * then we'll have saved the "is_dense" value and will be good to go.
+ *
+ * But non-upgraded thrift CF (and pre-7744 CF) will have no value for 
"is_dense", so we need
+ * to infer that information without relying on it in that case. And 
for the most part this is
+ * easy, a CF that has at least one REGULAR definition is not dense. 
But the subtlety is that not
+ * having a REGULAR definition may not mean dense because of CQL3 
definitions that have only the
+ * PRIMARY KEY defined.
+ *
+ * So we need to recognize those special case CQL3 table with only a 
primary key. If we have some
+ * clustering columns, we're fine as said above. So the only problem 
is that we cannot decide for
+ * sure if a CF without REGULAR columns nor CLUSTERING_COLUMN 
definition is meant to be dense, or if it
+ * has been created in CQL3 by say:
+ *CREATE TABLE test (k int PRIMARY KEY)
+ * in which case it should not be dense. However, we can limit our 
margin of error by assuming we are
+ * in the latter case only if the comparator is exactly 
CompositeType(UTF8Type).
+ */

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/020a80b1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/020a80b1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/020a80b1

Branch: refs/heads/cassandra-3.5
Commit: 020a80b132cfabc6032f0f96f477856b87712623
Parents: 03df4c7 1a1f836
Author: Aleksey Yeschenko 
Authored: Fri Apr 1 19:21:00 2016 +0100
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:21:00 2016 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/020a80b1/CHANGES.txt
--
diff --cc CHANGES.txt
index e94df56,0ce75a8..61265b1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.5
 +3.5
 + * Avoid index segment stitching in RAM which lead to OOM on big SSTable 
files (CASSANDRA-11383)
 + * Fix clustering and row filters for LIKE queries on clustering columns 
(CASSANDRA-11397)
 +Merged from 3.0:
+  * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
   * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
   * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
   * Upgrade ohc to 0.4.3

http://git-wip-us.apache.org/repos/asf/cassandra/blob/020a80b1/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--

[3/6] cassandra git commit: Fix rare NPE on schema upgrade from 2.x to 3.x

[aleksey]

Fix rare NPE on schema upgrade from 2.x to 3.x

patch by Aleksey Yeschenko; reviewed by Sylvain Lebresne for
CASSANDRA-10943


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a1f836f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a1f836f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a1f836f

Branch: refs/heads/trunk
Commit: 1a1f836f20ba4bf1c263d4fd3c32338e55abbcbc
Parents: 1513139
Author: Aleksey Yeschenko 
Authored: Wed Mar 23 21:21:29 2016 +
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:19:51 2016 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8d4834..0ce75a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
  * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--
diff --git a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java 
b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
index afa0f38..7411b93 100644
--- a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
+++ b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
@@ -284,7 +284,9 @@ public final class LegacySchemaMigrator
 AbstractType subComparator = tableRow.has("subcomparator") ? 
TypeParser.parse(tableRow.getString("subcomparator")) : null;
 
 boolean isSuper = 
"super".equals(tableRow.getString("type").toLowerCase());
-boolean isDense = tableRow.getBoolean("is_dense");
+boolean isDense = tableRow.has("is_dense")
+? tableRow.getBoolean("is_dense")
+: calculateIsDense(rawComparator, columnRows);
 boolean isCompound = rawComparator instanceof CompositeType;
 
 // We don't really use the default validator but as we have it for 
backward compatibility, we use it to know if it's a counter table
@@ -364,6 +366,65 @@ public final class LegacySchemaMigrator
   .triggers(createTriggersFromTriggerRows(triggerRows));
 }
 
+/*
+ * We call dense a CF for which each component of the comparator is a 
clustering column, i.e. no
+ * component is used to store a regular column names. In other words, 
non-composite static "thrift"
+ * and CQL3 CF are *not* dense.
+ * We save whether the table is dense or not during table creation through 
CQL, but we don't have this
+ * information for table just created through thrift, nor for table prior 
to CASSANDRA-7744, so this
+ * method does its best to infer whether the table is dense or not based 
on other elements.
+ */
+public static boolean calculateIsDense(AbstractType comparator, 
UntypedResultSet columnRows)
+{
+/*
+ * As said above, this method is only here because we need to deal 
with thrift upgrades.
+ * Once a CF has been "upgraded", i.e. we've rebuilt and save its CQL3 
metadata at least once,
+ * then we'll have saved the "is_dense" value and will be good to go.
+ *
+ * But non-upgraded thrift CF (and pre-7744 CF) will have no value for 
"is_dense", so we need
+ * to infer that information without relying on it in that case. And 
for the most part this is
+ * easy, a CF that has at least one REGULAR definition is not dense. 
But the subtlety is that not
+ * having a REGULAR definition may not mean dense because of CQL3 
definitions that have only the
+ * PRIMARY KEY defined.
+ *
+ * So we need to recognize those special case CQL3 table with only a 
primary key. If we have some
+ * clustering columns, we're fine as said above. So the only problem 
is that we cannot decide for
+ * sure if a CF without REGULAR columns nor CLUSTERING_COLUMN 
definition is meant to be dense, or if it
+ * has been created in CQL3 by say:
+ *CREATE TABLE test (k int PRIMARY KEY)
+ * in which case it should not be dense. However, we can limit our 
margin of error by assuming we are
+ * in the latter case only if the comparator is exactly 
CompositeType(UTF8Type).
+ */
+   

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/020a80b1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/020a80b1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/020a80b1

Branch: refs/heads/trunk
Commit: 020a80b132cfabc6032f0f96f477856b87712623
Parents: 03df4c7 1a1f836
Author: Aleksey Yeschenko 
Authored: Fri Apr 1 19:21:00 2016 +0100
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:21:00 2016 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/020a80b1/CHANGES.txt
--
diff --cc CHANGES.txt
index e94df56,0ce75a8..61265b1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.5
 +3.5
 + * Avoid index segment stitching in RAM which lead to OOM on big SSTable 
files (CASSANDRA-11383)
 + * Fix clustering and row filters for LIKE queries on clustering columns 
(CASSANDRA-11397)
 +Merged from 3.0:
+  * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
   * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
   * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
   * Upgrade ohc to 0.4.3

http://git-wip-us.apache.org/repos/asf/cassandra/blob/020a80b1/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--

[1/6] cassandra git commit: Fix rare NPE on schema upgrade from 2.x to 3.x

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 15131391d -> 1a1f836f2
  refs/heads/cassandra-3.5 03df4c746 -> 020a80b13
  refs/heads/trunk fc37e618b -> ee9fc22a6


Fix rare NPE on schema upgrade from 2.x to 3.x

patch by Aleksey Yeschenko; reviewed by Sylvain Lebresne for
CASSANDRA-10943


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a1f836f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a1f836f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a1f836f

Branch: refs/heads/cassandra-3.0
Commit: 1a1f836f20ba4bf1c263d4fd3c32338e55abbcbc
Parents: 1513139
Author: Aleksey Yeschenko 
Authored: Wed Mar 23 21:21:29 2016 +
Committer: Aleksey Yeschenko 
Committed: Fri Apr 1 19:19:51 2016 +0100

--
 CHANGES.txt |  1 +
 .../cassandra/schema/LegacySchemaMigrator.java  | 63 +++-
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d8d4834..0ce75a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Fix rare NPE on schema upgrade from 2.x to 3.x (CASSANDRA-10943)
  * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a1f836f/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--
diff --git a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java 
b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
index afa0f38..7411b93 100644
--- a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
+++ b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
@@ -284,7 +284,9 @@ public final class LegacySchemaMigrator
 AbstractType subComparator = tableRow.has("subcomparator") ? 
TypeParser.parse(tableRow.getString("subcomparator")) : null;
 
 boolean isSuper = 
"super".equals(tableRow.getString("type").toLowerCase());
-boolean isDense = tableRow.getBoolean("is_dense");
+boolean isDense = tableRow.has("is_dense")
+? tableRow.getBoolean("is_dense")
+: calculateIsDense(rawComparator, columnRows);
 boolean isCompound = rawComparator instanceof CompositeType;
 
 // We don't really use the default validator but as we have it for 
backward compatibility, we use it to know if it's a counter table
@@ -364,6 +366,65 @@ public final class LegacySchemaMigrator
   .triggers(createTriggersFromTriggerRows(triggerRows));
 }
 
+/*
+ * We call dense a CF for which each component of the comparator is a 
clustering column, i.e. no
+ * component is used to store a regular column names. In other words, 
non-composite static "thrift"
+ * and CQL3 CF are *not* dense.
+ * We save whether the table is dense or not during table creation through 
CQL, but we don't have this
+ * information for table just created through thrift, nor for table prior 
to CASSANDRA-7744, so this
+ * method does its best to infer whether the table is dense or not based 
on other elements.
+ */
+public static boolean calculateIsDense(AbstractType comparator, 
UntypedResultSet columnRows)
+{
+/*
+ * As said above, this method is only here because we need to deal 
with thrift upgrades.
+ * Once a CF has been "upgraded", i.e. we've rebuilt and save its CQL3 
metadata at least once,
+ * then we'll have saved the "is_dense" value and will be good to go.
+ *
+ * But non-upgraded thrift CF (and pre-7744 CF) will have no value for 
"is_dense", so we need
+ * to infer that information without relying on it in that case. And 
for the most part this is
+ * easy, a CF that has at least one REGULAR definition is not dense. 
But the subtlety is that not
+ * having a REGULAR definition may not mean dense because of CQL3 
definitions that have only the
+ * PRIMARY KEY defined.
+ *
+ * So we need to recognize those special case CQL3 table with only a 
primary key. If we have some
+ * clustering columns, we're fine as said above. So the only problem 
is that we cannot decide for
+ * sure if a CF without REGULAR columns nor CLUSTERING_COLUMN 
definition is meant to be dense, or if it
+ * has been created in CQL3 by say:
+ *CREATE TABLE test (k int PRIMARY KEY)
+ * in which case it 

[jira] [Commented] (CASSANDRA-11312) DatabaseDescriptor#applyConfig should log stacktrace in case of Eception during seed provider creation

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222080#comment-15222080
 ] 

Jason Brown commented on CASSANDRA-11312:
-

+1, will commit shortly.

> DatabaseDescriptor#applyConfig should log stacktrace in case of Eception 
> during seed provider creation
> --
>
> Key: CASSANDRA-11312
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11312
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Andrzej Ludwikowski
>Priority: Minor
> Attachments: 11312-2.2.txt
>
>
> The comment says that: "See log for stacktrace.", but with the the flag 
> "false" stacktrace is not logged. Logging stacktrace will save some time when 
> someone (like me) mess up with the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-11312) DatabaseDescriptor#applyConfig should log stacktrace in case of Eception during seed provider creation

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown reopened CASSANDRA-11312:
-

> DatabaseDescriptor#applyConfig should log stacktrace in case of Eception 
> during seed provider creation
> --
>
> Key: CASSANDRA-11312
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11312
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Andrzej Ludwikowski
>Priority: Minor
> Attachments: 11312-2.2.txt
>
>
> The comment says that: "See log for stacktrace.", but with the the flag 
> "false" stacktrace is not logged. Logging stacktrace will save some time when 
> someone (like me) mess up with the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11312) DatabaseDescriptor#applyConfig should log stacktrace in case of Eception during seed provider creation

[Andrzej Ludwikowski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Ludwikowski resolved CASSANDRA-11312.
-
Resolution: Fixed

> DatabaseDescriptor#applyConfig should log stacktrace in case of Eception 
> during seed provider creation
> --
>
> Key: CASSANDRA-11312
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11312
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Andrzej Ludwikowski
>Priority: Minor
> Attachments: 11312-2.2.txt
>
>
> The comment says that: "See log for stacktrace.", but with the the flag 
> "false" stacktrace is not logged. Logging stacktrace will save some time when 
> someone (like me) mess up with the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11461) Failed incremental repairs never cleared from pending list

[Nick Bailey (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222072#comment-15222072
 ] 

Nick Bailey commented on CASSANDRA-11461:
-

Hmm ok so we should prevent full incremental repairs from ever running on more 
than one node and never allow subrange incremental repairs?

> Failed incremental repairs never cleared from pending list
> --
>
> Key: CASSANDRA-11461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11461
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Adam Hattrell
>
> Set up a test cluster with 2 DC's, heavy use of LCS (not sure if that's 
> relevant).
> Kick off cassandra-stress against it.
> Kick of an automated incremental repair cycle.  
> After a bit a node starts flapping which causes a few repairs to fail.  This 
> is never cleared out of pending repairs - given the keyspace is replicated to 
> all nodes it means they all have pending repairs that will never complete.  
> Repairs  are basically blocked at this point.
> Given we're using Incremental repairs you're now spammed with:
> "Cannot start multiple repair sessions over the same sstables"
> Cluster and logs are still available for review - message me for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10587) sstablemetadata NPE on cassandra 2.2

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10587:
---
   Resolution: Fixed
Fix Version/s: (was: 2.2.x)
   (was: 3.x)
   3.5
   3.0.5
   2.2.6
   Status: Resolved  (was: Ready to Commit)

Committed as c662259fe9e1e25e10e58eb1146de80c53e69867, thanks!

> sstablemetadata NPE on cassandra 2.2
> 
>
> Key: CASSANDRA-10587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tiago Batista
>Assignee: Yuki Morishita
>Priority: Minor
> Fix For: 2.2.6, 3.0.5, 3.5
>
>
> I have recently upgraded my cassandra cluster to 2.2, currently running 
> 2.2.3. After running the first repair, cassandra renames the sstables to the 
> new naming schema that does not contain the keyspace name.
>  This causes sstablemetadata to fail with the following stack trace:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at 
> org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:275)
> at 
> org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:172)
> at 
> org.apache.cassandra.tools.SSTableMetadataViewer.main(SSTableMetadataViewer.java:52)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15131391
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15131391
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15131391

Branch: refs/heads/cassandra-3.0
Commit: 15131391d833c1a0ba0e26903e27867fd101fe72
Parents: 98086b6 c662259
Author: Yuki Morishita 
Authored: Fri Apr 1 12:44:52 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:44:52 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/CHANGES.txt
--
diff --cc CHANGES.txt
index 482c41a,c8a4f21..d8d4834
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,21 -1,9 +1,22 @@@
 -2.2.6
 +3.0.5
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
--
diff --cc src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
index 01bcb8a,000..3393b5c
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
@@@ -1,195 -1,0 +1,172 @@@
 +package org.apache.cassandra.db.lifecycle;
 +
 +import java.io.File;
 +import java.io.IOException;
 +import java.nio.file.DirectoryStream;
 +import java.nio.file.Files;
 +import java.nio.file.Path;
 +import java.util.*;
 +import java.util.function.BiFunction;
 +import java.util.stream.Collectors;
 +import java.util.stream.Stream;
 +import java.util.stream.StreamSupport;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +
 +import org.apache.cassandra.db.Directories;
 +
 +import static org.apache.cassandra.db.Directories.*;
 +
 +/**
 + * A class for listing files in a folder.
 + */
 +final class LogAwareFileLister
 +{
 +// The folder to scan
 +private final Path folder;
 +
 +// The filter determines which files the client wants returned
 +private final BiFunction filter; //file, file 
type
 +
 +// The behavior when we fail to list files
 +private final OnTxnErr onTxnErr;
 +
 +// The unfiltered result
 +NavigableMap files = new TreeMap<>();
 +
 +@VisibleForTesting
 +LogAwareFileLister(Path folder, BiFunction 
filter, OnTxnErr onTxnErr)
 +{
 +this.folder = folder;
 +this.filter = filter;
 +this.onTxnErr = onTxnErr;
 +}
 +
 +public List list()
 +{
 +try
 +{
 +return innerList();
 +}
 +catch (Throwable t)
 +{
 +throw new 

[02/10] cassandra git commit: Use canonical path for directory in SSTable descriptor

[yukim]

Use canonical path for directory in SSTable descriptor

patch by yukim; reviewed by Paulo Motta for CASSANDRA-10587


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c662259f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c662259f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c662259f

Branch: refs/heads/cassandra-3.0
Commit: c662259fe9e1e25e10e58eb1146de80c53e69867
Parents: caaa9fc
Author: Yuki Morishita 
Authored: Tue Feb 16 12:45:36 2016 -0600
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:10:08 2016 -0500

--
 CHANGES.txt |  1 +
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +---
 .../cassandra/io/sstable/DescriptorTest.java| 51 ++--
 3 files changed, 51 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 78ea961..c8a4f21 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 9f259fe..ed81616 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -18,6 +18,8 @@
 package org.apache.cassandra.io.sstable;
 
 import java.io.File;
+import java.io.IOError;
+import java.io.IOException;
 import java.util.ArrayDeque;
 import java.util.Deque;
 import java.util.StringTokenizer;
@@ -58,7 +60,7 @@ public class Descriptor
 }
 }
 
-
+/** canonicalized path to the directory where SSTable resides */
 public final File directory;
 /** version has the following format: [a-z]+ */
 public final Version version;
@@ -91,14 +93,21 @@ public class Descriptor
 {
 assert version != null && directory != null && ksname != null && 
cfname != null && 
formatType.info.getLatestVersion().getClass().equals(version.getClass());
 this.version = version;
-this.directory = directory;
+try
+{
+this.directory = directory.getCanonicalFile();
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
 this.ksname = ksname;
 this.cfname = cfname;
 this.generation = generation;
 this.type = temp;
 this.formatType = formatType;
 
-hashCode = Objects.hashCode(version, directory, generation, ksname, 
cfname, temp, formatType);
+hashCode = Objects.hashCode(version, this.directory, generation, 
ksname, cfname, temp, formatType);
 }
 
 public Descriptor withGeneration(int newGeneration)
@@ -168,8 +177,7 @@ public class Descriptor
  */
 public static Descriptor fromFilename(String filename)
 {
-File file = new File(filename);
-return fromFilename(file.getParentFile(), file.getName(), false).left;
+return fromFilename(filename, false);
 }
 
 public static Descriptor fromFilename(String filename, SSTableFormat.Type 
formatType)
@@ -179,7 +187,7 @@ public class Descriptor
 
 public static Descriptor fromFilename(String filename, boolean 
skipComponent)
 {
-File file = new File(filename);
+File file = new File(filename).getAbsoluteFile();
 return fromFilename(file.getParentFile(), file.getName(), 
skipComponent).left;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
--
diff --git a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java 
b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
index 70ab8ba..6354fc2 100644
--- a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
@@ -84,16 +84,22 @@ public class DescriptorTest
 // secondary index
 String idxName = "myidx";
 File idxDir = new File(dir.getAbsolutePath() + File.separator + 
Directories.SECONDARY_INDEX_NAME_SEPARATOR + idxName);
-checkFromFilename(new 

[04/10] cassandra git commit: Use canonical path for directory in SSTable descriptor

[yukim]

Use canonical path for directory in SSTable descriptor

patch by yukim; reviewed by Paulo Motta for CASSANDRA-10587


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c662259f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c662259f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c662259f

Branch: refs/heads/trunk
Commit: c662259fe9e1e25e10e58eb1146de80c53e69867
Parents: caaa9fc
Author: Yuki Morishita 
Authored: Tue Feb 16 12:45:36 2016 -0600
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:10:08 2016 -0500

--
 CHANGES.txt |  1 +
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +---
 .../cassandra/io/sstable/DescriptorTest.java| 51 ++--
 3 files changed, 51 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 78ea961..c8a4f21 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 9f259fe..ed81616 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -18,6 +18,8 @@
 package org.apache.cassandra.io.sstable;
 
 import java.io.File;
+import java.io.IOError;
+import java.io.IOException;
 import java.util.ArrayDeque;
 import java.util.Deque;
 import java.util.StringTokenizer;
@@ -58,7 +60,7 @@ public class Descriptor
 }
 }
 
-
+/** canonicalized path to the directory where SSTable resides */
 public final File directory;
 /** version has the following format: [a-z]+ */
 public final Version version;
@@ -91,14 +93,21 @@ public class Descriptor
 {
 assert version != null && directory != null && ksname != null && 
cfname != null && 
formatType.info.getLatestVersion().getClass().equals(version.getClass());
 this.version = version;
-this.directory = directory;
+try
+{
+this.directory = directory.getCanonicalFile();
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
 this.ksname = ksname;
 this.cfname = cfname;
 this.generation = generation;
 this.type = temp;
 this.formatType = formatType;
 
-hashCode = Objects.hashCode(version, directory, generation, ksname, 
cfname, temp, formatType);
+hashCode = Objects.hashCode(version, this.directory, generation, 
ksname, cfname, temp, formatType);
 }
 
 public Descriptor withGeneration(int newGeneration)
@@ -168,8 +177,7 @@ public class Descriptor
  */
 public static Descriptor fromFilename(String filename)
 {
-File file = new File(filename);
-return fromFilename(file.getParentFile(), file.getName(), false).left;
+return fromFilename(filename, false);
 }
 
 public static Descriptor fromFilename(String filename, SSTableFormat.Type 
formatType)
@@ -179,7 +187,7 @@ public class Descriptor
 
 public static Descriptor fromFilename(String filename, boolean 
skipComponent)
 {
-File file = new File(filename);
+File file = new File(filename).getAbsoluteFile();
 return fromFilename(file.getParentFile(), file.getName(), 
skipComponent).left;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
--
diff --git a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java 
b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
index 70ab8ba..6354fc2 100644
--- a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
@@ -84,16 +84,22 @@ public class DescriptorTest
 // secondary index
 String idxName = "myidx";
 File idxDir = new File(dir.getAbsolutePath() + File.separator + 
Directories.SECONDARY_INDEX_NAME_SEPARATOR + idxName);
-checkFromFilename(new 

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[yukim]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03df4c74
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03df4c74
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03df4c74

Branch: refs/heads/trunk
Commit: 03df4c7460f8047f4af2cfc1b19318394b68b788
Parents: 496418c 1513139
Author: Yuki Morishita 
Authored: Fri Apr 1 12:45:42 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:45:42 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/03df4c74/CHANGES.txt
--

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[yukim]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03df4c74
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03df4c74
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03df4c74

Branch: refs/heads/cassandra-3.5
Commit: 03df4c7460f8047f4af2cfc1b19318394b68b788
Parents: 496418c 1513139
Author: Yuki Morishita 
Authored: Fri Apr 1 12:45:42 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:45:42 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/03df4c74/CHANGES.txt
--

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15131391
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15131391
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15131391

Branch: refs/heads/trunk
Commit: 15131391d833c1a0ba0e26903e27867fd101fe72
Parents: 98086b6 c662259
Author: Yuki Morishita 
Authored: Fri Apr 1 12:44:52 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:44:52 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/CHANGES.txt
--
diff --cc CHANGES.txt
index 482c41a,c8a4f21..d8d4834
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,21 -1,9 +1,22 @@@
 -2.2.6
 +3.0.5
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
--
diff --cc src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
index 01bcb8a,000..3393b5c
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
@@@ -1,195 -1,0 +1,172 @@@
 +package org.apache.cassandra.db.lifecycle;
 +
 +import java.io.File;
 +import java.io.IOException;
 +import java.nio.file.DirectoryStream;
 +import java.nio.file.Files;
 +import java.nio.file.Path;
 +import java.util.*;
 +import java.util.function.BiFunction;
 +import java.util.stream.Collectors;
 +import java.util.stream.Stream;
 +import java.util.stream.StreamSupport;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +
 +import org.apache.cassandra.db.Directories;
 +
 +import static org.apache.cassandra.db.Directories.*;
 +
 +/**
 + * A class for listing files in a folder.
 + */
 +final class LogAwareFileLister
 +{
 +// The folder to scan
 +private final Path folder;
 +
 +// The filter determines which files the client wants returned
 +private final BiFunction filter; //file, file 
type
 +
 +// The behavior when we fail to list files
 +private final OnTxnErr onTxnErr;
 +
 +// The unfiltered result
 +NavigableMap files = new TreeMap<>();
 +
 +@VisibleForTesting
 +LogAwareFileLister(Path folder, BiFunction 
filter, OnTxnErr onTxnErr)
 +{
 +this.folder = folder;
 +this.filter = filter;
 +this.onTxnErr = onTxnErr;
 +}
 +
 +public List list()
 +{
 +try
 +{
 +return innerList();
 +}
 +catch (Throwable t)
 +{
 +throw new RuntimeException(String.format("Failed 

[10/10] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

[yukim]

Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fc37e618
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fc37e618
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fc37e618

Branch: refs/heads/trunk
Commit: fc37e618bb87971e5df82be1104ba7bec83e
Parents: 3efcdc5 03df4c7
Author: Yuki Morishita 
Authored: Fri Apr 1 12:45:48 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:45:48 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fc37e618/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fc37e618/test/unit/org/apache/cassandra/db/lifecycle/LogTransactionTest.java
--
diff --cc test/unit/org/apache/cassandra/db/lifecycle/LogTransactionTest.java
index 59958bb,45b5844..9defa8d
--- a/test/unit/org/apache/cassandra/db/lifecycle/LogTransactionTest.java
+++ b/test/unit/org/apache/cassandra/db/lifecycle/LogTransactionTest.java
@@@ -514,12 -515,12 +515,12 @@@ public class LogTransactionTest extend
  
  // test listing
  
Assert.assertEquals(sstables[0].getAllFilePaths().stream().map(File::new).collect(Collectors.toSet()),
- 
LogAwareFileLister.getTemporaryFiles(dataFolder1));
+ getTemporaryFiles(dataFolder1));
  
Assert.assertEquals(sstables[2].getAllFilePaths().stream().map(File::new).collect(Collectors.toSet()),
- 
LogAwareFileLister.getTemporaryFiles(dataFolder2));
+ getTemporaryFiles(dataFolder2));
  
  // normally called at startup
 -LogTransaction.removeUnfinishedLeftovers(Arrays.asList(dataFolder1, 
dataFolder2));
 +
assertTrue(LogTransaction.removeUnfinishedLeftovers(Arrays.asList(dataFolder1, 
dataFolder2)));
  
  // new tables should be only table left
  assertFiles(dataFolder1.getPath(), new 
HashSet<>(sstables[1].getAllFilePaths()));
@@@ -565,12 -566,12 +566,12 @@@
  
  // test listing
  
Assert.assertEquals(sstables[1].getAllFilePaths().stream().map(File::new).collect(Collectors.toSet()),
- 
LogAwareFileLister.getTemporaryFiles(dataFolder1));
+ getTemporaryFiles(dataFolder1));
  
Assert.assertEquals(sstables[3].getAllFilePaths().stream().map(File::new).collect(Collectors.toSet()),
- 
LogAwareFileLister.getTemporaryFiles(dataFolder2));
+ getTemporaryFiles(dataFolder2));
  
  // normally called at startup
 -LogTransaction.removeUnfinishedLeftovers(Arrays.asList(dataFolder1, 
dataFolder2));
 +
assertTrue(LogTransaction.removeUnfinishedLeftovers(Arrays.asList(dataFolder1, 
dataFolder2)));
  
  // old tables should be only table left
  assertFiles(dataFolder1.getPath(), new 
HashSet<>(sstables[0].getAllFilePaths()));

[01/10] cassandra git commit: Use canonical path for directory in SSTable descriptor

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 caaa9fc82 -> c662259fe
  refs/heads/cassandra-3.0 98086b65d -> 15131391d
  refs/heads/cassandra-3.5 496418ce0 -> 03df4c746
  refs/heads/trunk 3efcdc5f4 -> fc37e618b


Use canonical path for directory in SSTable descriptor

patch by yukim; reviewed by Paulo Motta for CASSANDRA-10587


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c662259f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c662259f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c662259f

Branch: refs/heads/cassandra-2.2
Commit: c662259fe9e1e25e10e58eb1146de80c53e69867
Parents: caaa9fc
Author: Yuki Morishita 
Authored: Tue Feb 16 12:45:36 2016 -0600
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:10:08 2016 -0500

--
 CHANGES.txt |  1 +
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +---
 .../cassandra/io/sstable/DescriptorTest.java| 51 ++--
 3 files changed, 51 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 78ea961..c8a4f21 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 9f259fe..ed81616 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -18,6 +18,8 @@
 package org.apache.cassandra.io.sstable;
 
 import java.io.File;
+import java.io.IOError;
+import java.io.IOException;
 import java.util.ArrayDeque;
 import java.util.Deque;
 import java.util.StringTokenizer;
@@ -58,7 +60,7 @@ public class Descriptor
 }
 }
 
-
+/** canonicalized path to the directory where SSTable resides */
 public final File directory;
 /** version has the following format: [a-z]+ */
 public final Version version;
@@ -91,14 +93,21 @@ public class Descriptor
 {
 assert version != null && directory != null && ksname != null && 
cfname != null && 
formatType.info.getLatestVersion().getClass().equals(version.getClass());
 this.version = version;
-this.directory = directory;
+try
+{
+this.directory = directory.getCanonicalFile();
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
 this.ksname = ksname;
 this.cfname = cfname;
 this.generation = generation;
 this.type = temp;
 this.formatType = formatType;
 
-hashCode = Objects.hashCode(version, directory, generation, ksname, 
cfname, temp, formatType);
+hashCode = Objects.hashCode(version, this.directory, generation, 
ksname, cfname, temp, formatType);
 }
 
 public Descriptor withGeneration(int newGeneration)
@@ -168,8 +177,7 @@ public class Descriptor
  */
 public static Descriptor fromFilename(String filename)
 {
-File file = new File(filename);
-return fromFilename(file.getParentFile(), file.getName(), false).left;
+return fromFilename(filename, false);
 }
 
 public static Descriptor fromFilename(String filename, SSTableFormat.Type 
formatType)
@@ -179,7 +187,7 @@ public class Descriptor
 
 public static Descriptor fromFilename(String filename, boolean 
skipComponent)
 {
-File file = new File(filename);
+File file = new File(filename).getAbsoluteFile();
 return fromFilename(file.getParentFile(), file.getName(), 
skipComponent).left;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
--
diff --git a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java 
b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
index 70ab8ba..6354fc2 100644
--- a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
@@ -84,16 +84,22 @@ public class 

[03/10] cassandra git commit: Use canonical path for directory in SSTable descriptor

[yukim]

Use canonical path for directory in SSTable descriptor

patch by yukim; reviewed by Paulo Motta for CASSANDRA-10587


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c662259f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c662259f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c662259f

Branch: refs/heads/cassandra-3.5
Commit: c662259fe9e1e25e10e58eb1146de80c53e69867
Parents: caaa9fc
Author: Yuki Morishita 
Authored: Tue Feb 16 12:45:36 2016 -0600
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:10:08 2016 -0500

--
 CHANGES.txt |  1 +
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +---
 .../cassandra/io/sstable/DescriptorTest.java| 51 ++--
 3 files changed, 51 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 78ea961..c8a4f21 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
  * Add cassandra-stress keystore option (CASSANDRA-9325)
  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 9f259fe..ed81616 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -18,6 +18,8 @@
 package org.apache.cassandra.io.sstable;
 
 import java.io.File;
+import java.io.IOError;
+import java.io.IOException;
 import java.util.ArrayDeque;
 import java.util.Deque;
 import java.util.StringTokenizer;
@@ -58,7 +60,7 @@ public class Descriptor
 }
 }
 
-
+/** canonicalized path to the directory where SSTable resides */
 public final File directory;
 /** version has the following format: [a-z]+ */
 public final Version version;
@@ -91,14 +93,21 @@ public class Descriptor
 {
 assert version != null && directory != null && ksname != null && 
cfname != null && 
formatType.info.getLatestVersion().getClass().equals(version.getClass());
 this.version = version;
-this.directory = directory;
+try
+{
+this.directory = directory.getCanonicalFile();
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
 this.ksname = ksname;
 this.cfname = cfname;
 this.generation = generation;
 this.type = temp;
 this.formatType = formatType;
 
-hashCode = Objects.hashCode(version, directory, generation, ksname, 
cfname, temp, formatType);
+hashCode = Objects.hashCode(version, this.directory, generation, 
ksname, cfname, temp, formatType);
 }
 
 public Descriptor withGeneration(int newGeneration)
@@ -168,8 +177,7 @@ public class Descriptor
  */
 public static Descriptor fromFilename(String filename)
 {
-File file = new File(filename);
-return fromFilename(file.getParentFile(), file.getName(), false).left;
+return fromFilename(filename, false);
 }
 
 public static Descriptor fromFilename(String filename, SSTableFormat.Type 
formatType)
@@ -179,7 +187,7 @@ public class Descriptor
 
 public static Descriptor fromFilename(String filename, boolean 
skipComponent)
 {
-File file = new File(filename);
+File file = new File(filename).getAbsoluteFile();
 return fromFilename(file.getParentFile(), file.getName(), 
skipComponent).left;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c662259f/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
--
diff --git a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java 
b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
index 70ab8ba..6354fc2 100644
--- a/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
@@ -84,16 +84,22 @@ public class DescriptorTest
 // secondary index
 String idxName = "myidx";
 File idxDir = new File(dir.getAbsolutePath() + File.separator + 
Directories.SECONDARY_INDEX_NAME_SEPARATOR + idxName);
-checkFromFilename(new 

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15131391
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15131391
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15131391

Branch: refs/heads/cassandra-3.5
Commit: 15131391d833c1a0ba0e26903e27867fd101fe72
Parents: 98086b6 c662259
Author: Yuki Morishita 
Authored: Fri Apr 1 12:44:52 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 1 12:44:52 2016 -0500

--
 CHANGES.txt |  1 +
 .../db/lifecycle/LogAwareFileLister.java| 23 ---
 .../apache/cassandra/io/sstable/Descriptor.java | 20 +--
 .../db/lifecycle/LogTransactionTest.java| 63 +++-
 .../cassandra/io/sstable/DescriptorTest.java| 40 +
 5 files changed, 90 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/CHANGES.txt
--
diff --cc CHANGES.txt
index 482c41a,c8a4f21..d8d4834
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,21 -1,9 +1,22 @@@
 -2.2.6
 +3.0.5
 + * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
 + * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
 + * Upgrade ohc to 0.4.3
 + * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)
 + * Allocate merkletrees with the correct size (CASSANDRA-11390)
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Use canonical path for directory in SSTable descriptor (CASSANDRA-10587)
   * Add cassandra-stress keystore option (CASSANDRA-9325)
 - * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 - * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
   * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
   * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
   * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/15131391/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
--
diff --cc src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
index 01bcb8a,000..3393b5c
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java
@@@ -1,195 -1,0 +1,172 @@@
 +package org.apache.cassandra.db.lifecycle;
 +
 +import java.io.File;
 +import java.io.IOException;
 +import java.nio.file.DirectoryStream;
 +import java.nio.file.Files;
 +import java.nio.file.Path;
 +import java.util.*;
 +import java.util.function.BiFunction;
 +import java.util.stream.Collectors;
 +import java.util.stream.Stream;
 +import java.util.stream.StreamSupport;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +
 +import org.apache.cassandra.db.Directories;
 +
 +import static org.apache.cassandra.db.Directories.*;
 +
 +/**
 + * A class for listing files in a folder.
 + */
 +final class LogAwareFileLister
 +{
 +// The folder to scan
 +private final Path folder;
 +
 +// The filter determines which files the client wants returned
 +private final BiFunction filter; //file, file 
type
 +
 +// The behavior when we fail to list files
 +private final OnTxnErr onTxnErr;
 +
 +// The unfiltered result
 +NavigableMap files = new TreeMap<>();
 +
 +@VisibleForTesting
 +LogAwareFileLister(Path folder, BiFunction 
filter, OnTxnErr onTxnErr)
 +{
 +this.folder = folder;
 +this.filter = filter;
 +this.onTxnErr = onTxnErr;
 +}
 +
 +public List list()
 +{
 +try
 +{
 +return innerList();
 +}
 +catch (Throwable t)
 +{
 +throw new 

[2/2] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[tylerhobbs]

Merge branch 'cassandra-3.0' into cassandra-3.5

Conflicts:
CHANGES.txt
pylib/cqlshlib/copyutil.py


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/496418ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/496418ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/496418ce

Branch: refs/heads/cassandra-3.5
Commit: 496418ce07705010c41b6674a564d90d9a2402bc
Parents: af9b9cd 98086b6
Author: Tyler Hobbs 
Authored: Fri Apr 1 12:32:01 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:32:01 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/CHANGES.txt
--
diff --cc CHANGES.txt
index 73d9ab5,482c41a..35e4884
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.5
 +3.5
 + * Avoid index segment stitching in RAM which lead to OOM on big SSTable 
files (CASSANDRA-11383)
 + * Fix clustering and row filters for LIKE queries on clustering columns 
(CASSANDRA-11397)
 +Merged from 3.0:
+  * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
   * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
   * Upgrade ohc to 0.4.3
   * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/bin/cqlsh.py
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/conf/cqlshrc.sample
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/pylib/cqlshlib/copyutil.py
--
diff --cc pylib/cqlshlib/copyutil.py
index 0cae396,53bbe09..3dd7485
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -278,8 -329,13 +329,14 @@@ class CopyTask(object)
  copy_options['ratefile'] = safe_normpath(opts.pop('ratefile', ''))
  copy_options['maxoutputsize'] = int(opts.pop('maxoutputsize', '-1'))
  copy_options['preparedstatements'] = 
bool(opts.pop('preparedstatements', 'true').lower() == 'true')
 +copy_options['ttl'] = int(opts.pop('ttl', -1))
  
+ # Hidden properties, they do not appear in the documentation but can 
be set in config files
+ # or on the cmd line but w/o completion
+ copy_options['maxinflightmessages'] = 
int(opts.pop('maxinflightmessages', '512'))
+ copy_options['maxbackoffattempts'] = 
int(opts.pop('maxbackoffattempts', '12'))
+ copy_options['maxpendingchunks'] = int(opts.pop('maxpendingchunks', 
'24'))
+ 
  self.check_options(copy_options)
  return CopyOptions(copy=copy_options, dialect=dialect_options, 
unrecognized=opts)
  
@@@ -2033,7 -2087,9 +2091,10 @@@ class ImportProcess(ChildProcess)
  self.min_batch_size = options.copy['minbatchsize']
  self.max_batch_size = options.copy['maxbatchsize']
  self.use_prepared_statements = options.copy['preparedstatements']
 +self.ttl = options.copy['ttl']
+ self.max_inflight_messages = options.copy['maxinflightmessages']
+ self.max_backoff_attempts = options.copy['maxbackoffattempts']
+ 
  self.dialect_options = options.dialect
  self._session = None
  self.query = None

[1/3] cassandra git commit: cqlsh: Improve backoff policy for COPY FROM

[tylerhobbs]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 6d1254ac2 -> 3efcdc5f4


cqlsh: Improve backoff policy for COPY FROM

Patch by Stefania Alborghetti; reviewed by Tyler Hobbs for
CASSANDRA-11320


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98086b65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98086b65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98086b65

Branch: refs/heads/trunk
Commit: 98086b65d0bc76631a3aeb50cddd8c9a82bc05b9
Parents: bd4cab2
Author: Stefania Alborghetti 
Authored: Fri Apr 1 12:29:01 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:29:01 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b9376bc..482c41a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3
  * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index e4ed830..1ef95ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -1886,7 +1886,7 @@ class Shell(cmd.Cmd):
   SKIPROWS=0  - the number of rows to skip
   SKIPCOLS='' - a comma separated list of column names to 
skip
   MAXPARSEERRORS=-1   - the maximum global number of parsing 
errors, -1 means no maximum
-  MAXINSERTERRORS=-1  - the maximum global number of insert 
errors, -1 means no maximum
+  MAXINSERTERRORS=1000- the maximum global number of insert 
errors, -1 means no maximum
   ERRFILE=''  - a file where to store all rows that could 
not be imported, by default this is
 import_ks_table.err where  is your 
keyspace and  is your table name.
   PREPAREDSTATEMENTS=True - whether to use prepared statements when 
importing, by default True. Set this to

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index 462dcc6..cb02b04 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -202,7 +202,7 @@ port = 9042
 ; maxparseerrors = -1
 
 ;; The maximum global number of insert errors to ignore, -1 means unlimited
-; maxinserterrors = -1
+; maxinserterrors = 1000
 
 ;; A file to store all rows that could not be imported, by default this is
 ;; import__.err where  is your keyspace and  is your 
table name.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 7aa5a96..53bbe09 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -28,8 +28,8 @@ import random
 import re
 import struct
 import sys
-import time
 import threading
+import time
 import traceback
 
 from bisect import bisect_right
@@ -37,16 +37,16 @@ from calendar import timegm
 from collections import defaultdict, namedtuple
 from decimal import Decimal
 from Queue import Queue
-from random import randrange
+from random import randint
 from StringIO import StringIO
 from select import select
 from uuid import UUID
 from util import profile_on, profile_off
 
-from cassandra.cluster import Cluster
+from cassandra.cluster import Cluster, DefaultConnection
 from cassandra.cqltypes import ReversedType, UserType
 from cassandra.metadata import protect_name, protect_names, protect_value
-from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy
+from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy, FallthroughRetryPolicy
 from cassandra.query import BatchStatement, BatchType, SimpleStatement, 
tuple_factory
 from cassandra.util import Date, Time
 
@@ -57,6 +57,7 @@ from sslhandling import ssl_settings
 
 PROFILE_ON = False
 STRACE_ON = False
+DEBUG = False  # This may be set to True when initializing the task
 IS_LINUX = platform.system() == 'Linux'
 
 CopyOptions = 

[3/3] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

[tylerhobbs]

Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3efcdc5f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3efcdc5f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3efcdc5f

Branch: refs/heads/trunk
Commit: 3efcdc5f458de8a7dc5e89cfe4718787ee86142a
Parents: 6d1254a 496418c
Author: Tyler Hobbs 
Authored: Fri Apr 1 12:36:42 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:36:42 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3efcdc5f/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3efcdc5f/bin/cqlsh.py
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3efcdc5f/conf/cqlshrc.sample
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3efcdc5f/pylib/cqlshlib/copyutil.py
--

[2/3] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[tylerhobbs]

Merge branch 'cassandra-3.0' into cassandra-3.5

Conflicts:
CHANGES.txt
pylib/cqlshlib/copyutil.py


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/496418ce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/496418ce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/496418ce

Branch: refs/heads/trunk
Commit: 496418ce07705010c41b6674a564d90d9a2402bc
Parents: af9b9cd 98086b6
Author: Tyler Hobbs 
Authored: Fri Apr 1 12:32:01 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:32:01 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/CHANGES.txt
--
diff --cc CHANGES.txt
index 73d9ab5,482c41a..35e4884
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.5
 +3.5
 + * Avoid index segment stitching in RAM which lead to OOM on big SSTable 
files (CASSANDRA-11383)
 + * Fix clustering and row filters for LIKE queries on clustering columns 
(CASSANDRA-11397)
 +Merged from 3.0:
+  * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
   * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
   * Upgrade ohc to 0.4.3
   * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/bin/cqlsh.py
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/conf/cqlshrc.sample
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/496418ce/pylib/cqlshlib/copyutil.py
--
diff --cc pylib/cqlshlib/copyutil.py
index 0cae396,53bbe09..3dd7485
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -278,8 -329,13 +329,14 @@@ class CopyTask(object)
  copy_options['ratefile'] = safe_normpath(opts.pop('ratefile', ''))
  copy_options['maxoutputsize'] = int(opts.pop('maxoutputsize', '-1'))
  copy_options['preparedstatements'] = 
bool(opts.pop('preparedstatements', 'true').lower() == 'true')
 +copy_options['ttl'] = int(opts.pop('ttl', -1))
  
+ # Hidden properties, they do not appear in the documentation but can 
be set in config files
+ # or on the cmd line but w/o completion
+ copy_options['maxinflightmessages'] = 
int(opts.pop('maxinflightmessages', '512'))
+ copy_options['maxbackoffattempts'] = 
int(opts.pop('maxbackoffattempts', '12'))
+ copy_options['maxpendingchunks'] = int(opts.pop('maxpendingchunks', 
'24'))
+ 
  self.check_options(copy_options)
  return CopyOptions(copy=copy_options, dialect=dialect_options, 
unrecognized=opts)
  
@@@ -2033,7 -2087,9 +2091,10 @@@ class ImportProcess(ChildProcess)
  self.min_batch_size = options.copy['minbatchsize']
  self.max_batch_size = options.copy['maxbatchsize']
  self.use_prepared_statements = options.copy['preparedstatements']
 +self.ttl = options.copy['ttl']
+ self.max_inflight_messages = options.copy['maxinflightmessages']
+ self.max_backoff_attempts = options.copy['maxbackoffattempts']
+ 
  self.dialect_options = options.dialect
  self._session = None
  self.query = None

cassandra git commit: cqlsh: Improve backoff policy for COPY FROM

[tylerhobbs]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 bd4cab24b -> 98086b65d


cqlsh: Improve backoff policy for COPY FROM

Patch by Stefania Alborghetti; reviewed by Tyler Hobbs for
CASSANDRA-11320


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98086b65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98086b65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98086b65

Branch: refs/heads/cassandra-3.0
Commit: 98086b65d0bc76631a3aeb50cddd8c9a82bc05b9
Parents: bd4cab2
Author: Stefania Alborghetti 
Authored: Fri Apr 1 12:29:01 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:29:01 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b9376bc..482c41a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3
  * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index e4ed830..1ef95ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -1886,7 +1886,7 @@ class Shell(cmd.Cmd):
   SKIPROWS=0  - the number of rows to skip
   SKIPCOLS='' - a comma separated list of column names to 
skip
   MAXPARSEERRORS=-1   - the maximum global number of parsing 
errors, -1 means no maximum
-  MAXINSERTERRORS=-1  - the maximum global number of insert 
errors, -1 means no maximum
+  MAXINSERTERRORS=1000- the maximum global number of insert 
errors, -1 means no maximum
   ERRFILE=''  - a file where to store all rows that could 
not be imported, by default this is
 import_ks_table.err where  is your 
keyspace and  is your table name.
   PREPAREDSTATEMENTS=True - whether to use prepared statements when 
importing, by default True. Set this to

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index 462dcc6..cb02b04 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -202,7 +202,7 @@ port = 9042
 ; maxparseerrors = -1
 
 ;; The maximum global number of insert errors to ignore, -1 means unlimited
-; maxinserterrors = -1
+; maxinserterrors = 1000
 
 ;; A file to store all rows that could not be imported, by default this is
 ;; import__.err where  is your keyspace and  is your 
table name.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 7aa5a96..53bbe09 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -28,8 +28,8 @@ import random
 import re
 import struct
 import sys
-import time
 import threading
+import time
 import traceback
 
 from bisect import bisect_right
@@ -37,16 +37,16 @@ from calendar import timegm
 from collections import defaultdict, namedtuple
 from decimal import Decimal
 from Queue import Queue
-from random import randrange
+from random import randint
 from StringIO import StringIO
 from select import select
 from uuid import UUID
 from util import profile_on, profile_off
 
-from cassandra.cluster import Cluster
+from cassandra.cluster import Cluster, DefaultConnection
 from cassandra.cqltypes import ReversedType, UserType
 from cassandra.metadata import protect_name, protect_names, protect_value
-from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy
+from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy, FallthroughRetryPolicy
 from cassandra.query import BatchStatement, BatchType, SimpleStatement, 
tuple_factory
 from cassandra.util import Date, Time
 
@@ -57,6 +57,7 @@ from sslhandling import ssl_settings
 
 PROFILE_ON = False
 STRACE_ON = False
+DEBUG = False  # This may be set to True when initializing the task
 IS_LINUX = platform.system() == 'Linux'
 
 

[1/2] cassandra git commit: cqlsh: Improve backoff policy for COPY FROM

[tylerhobbs]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.5 af9b9cd5a -> 496418ce0


cqlsh: Improve backoff policy for COPY FROM

Patch by Stefania Alborghetti; reviewed by Tyler Hobbs for
CASSANDRA-11320


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98086b65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98086b65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98086b65

Branch: refs/heads/cassandra-3.5
Commit: 98086b65d0bc76631a3aeb50cddd8c9a82bc05b9
Parents: bd4cab2
Author: Stefania Alborghetti 
Authored: Fri Apr 1 12:29:01 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 1 12:29:01 2016 -0500

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   |   2 +-
 conf/cqlshrc.sample|   2 +-
 pylib/cqlshlib/copyutil.py | 267 +---
 4 files changed, 168 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b9376bc..482c41a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Improve backoff policy for cqlsh COPY FROM (CASSANDRA-11320)
  * Improve IF NOT EXISTS check in CREATE INDEX (CASSANDRA-11131)
  * Upgrade ohc to 0.4.3
  * Enable SO_REUSEADDR for JMX RMI server sockets (CASSANDRA-11093)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index e4ed830..1ef95ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -1886,7 +1886,7 @@ class Shell(cmd.Cmd):
   SKIPROWS=0  - the number of rows to skip
   SKIPCOLS='' - a comma separated list of column names to 
skip
   MAXPARSEERRORS=-1   - the maximum global number of parsing 
errors, -1 means no maximum
-  MAXINSERTERRORS=-1  - the maximum global number of insert 
errors, -1 means no maximum
+  MAXINSERTERRORS=1000- the maximum global number of insert 
errors, -1 means no maximum
   ERRFILE=''  - a file where to store all rows that could 
not be imported, by default this is
 import_ks_table.err where  is your 
keyspace and  is your table name.
   PREPAREDSTATEMENTS=True - whether to use prepared statements when 
importing, by default True. Set this to

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index 462dcc6..cb02b04 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -202,7 +202,7 @@ port = 9042
 ; maxparseerrors = -1
 
 ;; The maximum global number of insert errors to ignore, -1 means unlimited
-; maxinserterrors = -1
+; maxinserterrors = 1000
 
 ;; A file to store all rows that could not be imported, by default this is
 ;; import__.err where  is your keyspace and  is your 
table name.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98086b65/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 7aa5a96..53bbe09 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -28,8 +28,8 @@ import random
 import re
 import struct
 import sys
-import time
 import threading
+import time
 import traceback
 
 from bisect import bisect_right
@@ -37,16 +37,16 @@ from calendar import timegm
 from collections import defaultdict, namedtuple
 from decimal import Decimal
 from Queue import Queue
-from random import randrange
+from random import randint
 from StringIO import StringIO
 from select import select
 from uuid import UUID
 from util import profile_on, profile_off
 
-from cassandra.cluster import Cluster
+from cassandra.cluster import Cluster, DefaultConnection
 from cassandra.cqltypes import ReversedType, UserType
 from cassandra.metadata import protect_name, protect_names, protect_value
-from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy
+from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
DCAwareRoundRobinPolicy, FallthroughRetryPolicy
 from cassandra.query import BatchStatement, BatchType, SimpleStatement, 
tuple_factory
 from cassandra.util import Date, Time
 
@@ -57,6 +57,7 @@ from sslhandling import ssl_settings
 
 PROFILE_ON = False
 STRACE_ON = False
+DEBUG = False  # This may be set to True when initializing the task
 IS_LINUX = platform.system() == 'Linux'
 
 

[jira] [Updated] (CASSANDRA-11475) MV code refactor

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11475:

Reviewer: Carl Yeksigian

> MV code refactor
> 
>
> Key: CASSANDRA-11475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11475
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.x
>
>
> While working on CASSANDRA-5546 I run into a problem with TTLs on MVs, which 
> looking more closely is a bug of the MV code. But one thing leading to 
> another I reviewed a good portion of the MV code and found the following 
> correction problems:
> * If a base row is TTLed then even if an update remove that TTL the view 
> entry remained TTLed and expires, leading to an inconsistency.
> * Due to calling the wrong ctor for {{LivenessInfo}}, when a TTL was set on 
> the base table, the view entry was living twice as long as the TTL. Again 
> leading to a temporary inconsistency.
> * When reading existing data to compute view updates, all deletion 
> informations are completely ignored (the code uses a {{PartitionIterator}} 
> instead of an {{UnfilteredPartitionIterator}}). This is a serious issue since 
> it means some deletions could be totally ignored as far as views are 
> concerned especially when messages are delivered to a replica out of order. 
> I'll note that while the 2 previous points are relatively easy to fix, I 
> didn't find an easy and clean way to fix this one on the current code.
> Further, I think the MV code in general has inefficiencies/code complexities 
> that should be avoidable:
> * {{TemporalRow.Set}} is buffering both everything read and a pretty much 
> complete copy of the updates. That's a potentially high memory requirement. 
> We shouldn't have to copy the updates and we shouldn't buffer all reads but 
> rather work incrementally.
> * {{TemporalRow}}/{{TemporalRow.Set}}/{{TemporalCell}} classes are somewhat 
> re-inventing the wheel. They are really just storing both an update we're 
> doing and the corresponding existing data, but we already have 
> {{Row}}/{{Partition}}/{{Cell}} for that. In practice, those {{Temporal*}} 
> class generates a lot of allocations that we could avoid.
> * The code from CASSANDRA-10060 to avoid multiple reads of the base table 
> with multiple views doesn't work when the update has partition/range 
> tombstones because the code uses {{TemporalRow.Set.setTombstonedExisting()}} 
> to trigger reuse, but the {{TemporalRow.Set.withNewViewPrimaryKey()}} method 
> is used between view and it does not preseve the {{hasTombstonedExisting}} 
> flag.  But that oversight, which is trivial to fix, is kind of a good thing 
> since if you fix it, you're left with a correction problem.
>   The read done when there is a partition deletion depends on the view itself 
> (if there is clustering filters in particular) and so reusing that read for 
> other views is wrong. Which makes that whole reuse code really dodgy imo: the 
> read for existing data is in {{View.java}}, suggesting that it depends on the 
> view (which again, it does at least for partition deletion), but it shouldn't 
> if we're going to reuse the result across multiple views.
> * Even ignoring the previous point, we still potentially read the base table 
> twice if the update mix both row updates and partition/range deletions, 
> potentially re-reading the same values.
> * It's probably more minor but the reading code is using {{QueryPager}}, 
> which is probably an artifact of the initial version of the code being 
> pre-8099, but it's not necessary anymore (the reads are local and locally 
> we're completely iterator based), adding, especially when we do page. I'll 
> note that despite using paging, the current code still buffers everything in 
> {{TemporalRow.Set}} anyway .
> Overall, I suspect trying to fix the problems above (particularly the fact 
> that existing deletion infos are ignored) is only going to add complexity 
> with the current code and we'd still have to fix the inefficiencies. So I 
> propose a refactor of that code which does 2 main things:
> # it removes all of {{TemporalRow}} and related classes. Instead, it directly 
> uses the existing {{Row}} (with all its deletion infos) and the update being 
> applied to it and compute the updates for the view from that. I submit that 
> this is more clear/simple, but this also avoid copying every cell of both the 
> existing and update data as a {{TemporalCell}}. We can also reuse codes like 
> {{Rows.merge}} and {{Rows.diff}} to make the handling of deletions relatively 
> painless.
> # instead of dealing with each view one at a time, re-iterating over all 
> updates each time, it iterates over each individual updates once and deal 
> with each view for that 

[jira] [Created] (CASSANDRA-11478) cassandra-stress reports average garbage collection time NaN ms.

[Cameron MacMinn (JIRA)]

Cameron MacMinn created CASSANDRA-11478:
---

 Summary: cassandra-stress reports average garbage collection time 
NaN ms.
 Key: CASSANDRA-11478
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11478
 Project: Cassandra
  Issue Type: Bug
  Components: Testing
 Environment: cassandra-stress 2.1.13
Reporter: Cameron MacMinn
Priority: Minor


As a tester, I prefer it say 'avg gc time(ms): 0'.

To reproduce problem, run default test from docs with a couple of extra options:

$ cassandra-stress write n=100 -node $(hostname) -mode native cql3 user=... 
password=... -log interval=1

My run said:

Results:
op rate   : 36054 [WRITE:36054]
partition rate: 36054 [WRITE:36054]
row rate  : 36054 [WRITE:36054]
latency mean  : 5.4 [WRITE:5.4]
latency median: 2.8 [WRITE:2.8]
latency 95th percentile   : 7.8 [WRITE:7.8]
latency 99th percentile   : 13.6 [WRITE:13.6]
latency 99.9th percentile : 70.7 [WRITE:70.7]
latency max   : 2691.2 [WRITE:2691.2]
Total partitions  : 100 [WRITE:100]
Total errors  : 0 [WRITE:0]
total gc count: 0
total gc mb   : 0
total gc time (s) : 0
avg gc time(ms)   : NaN
stdev gc time(ms) : 0
Total operation time  : 00:00:27
END

Also saw NaN in unrelated Jira bug report, while searching to see if this bug 
was reported yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11467) Paging loses rows in certain conditions

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11467:

Status: Ready to Commit  (was: Patch Available)

> Paging loses rows in certain conditions
> ---
>
> Key: CASSANDRA-11467
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11467
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Ian McMahon
>Assignee: Benjamin Lerer
> Fix For: 2.2.x
>
> Attachments: 11467-2.2.txt, pagination_test.go
>
>
> The bug occurs under the following conditions:
>   - RandomPartitioner
>   - a compact storage CF
>   - querying across rows
>   - a tombstone in the first column of a row on the pagesize boundary
>   - you need to be querying at least 2*pagesize + 1 records
> Attached is a go program using gocql which reproduces the bug fairly 
> minimally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11467) Paging loses rows in certain conditions

[Tyler Hobbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221958#comment-15221958
 ] 

Tyler Hobbs commented on CASSANDRA-11467:
-

+1

If 2.1 is affected, I think we should commit this there as well.  This isn't a 
very common query path to hit, but it's a fairly serious bug to have missing 
query results.

> Paging loses rows in certain conditions
> ---
>
> Key: CASSANDRA-11467
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11467
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Ian McMahon
>Assignee: Benjamin Lerer
> Fix For: 2.2.x
>
> Attachments: 11467-2.2.txt, pagination_test.go
>
>
> The bug occurs under the following conditions:
>   - RandomPartitioner
>   - a compact storage CF
>   - querying across rows
>   - a tombstone in the first column of a row on the pagesize boundary
>   - you need to be querying at least 2*pagesize + 1 records
> Attached is a go program using gocql which reproduces the bug fairly 
> minimally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Fabien Rousseau (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221919#comment-15221919
 ] 

Fabien Rousseau commented on CASSANDRA-11349:
-

I tested against the 3.0.4 and it is not affected (not tested the 3.X, but 
assumed that it's not affected).
There is another similar ticket (the 3.0.4 is not affected): 
https://issues.apache.org/jira/browse/CASSANDRA-11477 

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11051) Make LZ4 Compression Level Configurable

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11051:
--
Status: Ready to Commit  (was: Patch Available)

> Make LZ4 Compression Level Configurable 
> 
>
> Key: CASSANDRA-11051
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11051
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: lz4_2.2.patch
>
>
> We'd like to make the LZ4 Compressor implementation configurable on a per 
> column family basis. Testing has shown a ~4% reduction in file size with the 
> higher compression LZ4 implementation vs the standard compressor we currently 
> use instantiated by the default constructor. The attached patch adds the 
> following optional parameters 'lz4_compressor_type' and 
> 'lz4_high_compressor_level' to the LZ4Compressor. If none of the new optional 
> parameters are specified, the Compressor will use the same defaults Cassandra 
> has always had for LZ4.
> New LZ4Compressor Optional Parameters:
>   * lz4_compressor_type can currently be either 'high' (uses LZ4HCCompressor) 
> or 'fast' (uses LZ4Compressor)
>   * lz4_high_compressor_level can be set between 1 and 17. Not specifying a 
> compressor level while specifying lz4_compressor_type as 'high' will use a 
> default level of 9 (as picked by the LZ4 library as the "default").
> Currently, we use the default LZ4 compressor constructor. This change would 
> just expose the level (and implementation to use) to the user via the schema. 
> There are many potential cases where users may find that the tradeoff in 
> additional CPU and memory usage is worth the on-disk space savings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11051) Make LZ4 Compression Level Configurable

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221910#comment-15221910
 ] 

Michael Kjellman commented on CASSANDRA-11051:
--

Sorry [~krummas] I missed your updates! Rebase looks great, thanks for doing 
that. Ship it!

> Make LZ4 Compression Level Configurable 
> 
>
> Key: CASSANDRA-11051
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11051
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: lz4_2.2.patch
>
>
> We'd like to make the LZ4 Compressor implementation configurable on a per 
> column family basis. Testing has shown a ~4% reduction in file size with the 
> higher compression LZ4 implementation vs the standard compressor we currently 
> use instantiated by the default constructor. The attached patch adds the 
> following optional parameters 'lz4_compressor_type' and 
> 'lz4_high_compressor_level' to the LZ4Compressor. If none of the new optional 
> parameters are specified, the Compressor will use the same defaults Cassandra 
> has always had for LZ4.
> New LZ4Compressor Optional Parameters:
>   * lz4_compressor_type can currently be either 'high' (uses LZ4HCCompressor) 
> or 'fast' (uses LZ4Compressor)
>   * lz4_high_compressor_level can be set between 1 and 17. Not specifying a 
> compressor level while specifying lz4_compressor_type as 'high' will use a 
> default level of 9 (as picked by the LZ4 library as the "default").
> Currently, we use the default LZ4 compressor constructor. This change would 
> just expose the level (and implementation to use) to the user via the schema. 
> There are many potential cases where users may find that the tradeoff in 
> additional CPU and memory usage is worth the on-disk space savings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

svn commit: r1737413 - in /cassandra/site: publish/index.html src/content/index.html

[jbellis]

Author: jbellis
Date: Fri Apr  1 16:05:32 2016
New Revision: 1737413

URL: http://svn.apache.org/viewvc?rev=1737413=rev
Log:
formatting

Modified:
cassandra/site/publish/index.html
cassandra/site/src/content/index.html

Modified: cassandra/site/publish/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/publish/index.html?rev=1737413=1737412=1737413=diff
==
--- cassandra/site/publish/index.html (original)
+++ cassandra/site/publish/index.html Fri Apr  1 16:05:32 2016
@@ -180,7 +180,8 @@
   
 Discussion
 
-  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>Questions about 
using Cassandra.
+  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>QA about using 
Cassandra.
+  
   
   Users list  General mailing list for users.
   mailto:user-subscr...@cassandra.apache.org;>Subscribe

Modified: cassandra/site/src/content/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/src/content/index.html?rev=1737413=1737412=1737413=diff
==
--- cassandra/site/src/content/index.html (original)
+++ cassandra/site/src/content/index.html Fri Apr  1 16:05:32 2016
@@ -118,7 +118,8 @@
   
 Discussion
 
-  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>Questions about 
using Cassandra.
+  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>QA about using 
Cassandra.
+  
   
   Users list  General mailing list for users.
   mailto:user-subscr...@cassandra.apache.org;>Subscribe

svn commit: r1737412 - in /cassandra/site: publish/index.html src/content/index.html

[jbellis]

Author: jbellis
Date: Fri Apr  1 16:04:34 2016
New Revision: 1737412

URL: http://svn.apache.org/viewvc?rev=1737412=rev
Log:
add Stack Overflow; remove client dev list

Modified:
cassandra/site/publish/index.html
cassandra/site/src/content/index.html

Modified: cassandra/site/publish/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/publish/index.html?rev=1737412=1737411=1737412=diff
==
--- cassandra/site/publish/index.html (original)
+++ cassandra/site/publish/index.html Fri Apr  1 16:04:34 2016
@@ -178,27 +178,24 @@
 
   
   
-Mailing Lists
+Discussion
 
-  Users  General discussion list for users.
+  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>Questions about 
using Cassandra.
+  
+  Users list  General mailing list for users.
   mailto:user-subscr...@cassandra.apache.org;>Subscribe
   | mailto:user-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/user@cassandra.apache.org/;>Archives
   
-  Developers  Development related discussion.
+  Developers list  Development related mailing list.
   mailto:dev-subscr...@cassandra.apache.org;>Subscribe
   | mailto:dev-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/dev@cassandra.apache.org/;>Archives
   
-  Commits  Commit notification source repository.
+  Commits list  Commit notifications.
   mailto:commits-subscr...@cassandra.apache.org;>Subscribe
   | mailto:commits-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/commits@cassandra.apache.org/;>Archives
-  Client Libraries  Discussion related to the development
-of idiomatic client APIs.
-  mailto:client-dev-subscr...@cassandra.apache.org;>Subscribe
-  | mailto:client-dev-unsubscr...@cassandra.apache.org;>Unsubscribe
-  | http://www.mail-archive.com/client-dev@cassandra.apache.org/;>Archives
   
 
   

Modified: cassandra/site/src/content/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/src/content/index.html?rev=1737412=1737411=1737412=diff
==
--- cassandra/site/src/content/index.html (original)
+++ cassandra/site/src/content/index.html Fri Apr  1 16:04:34 2016
@@ -116,27 +116,24 @@
 
   
   
-Mailing Lists
+Discussion
 
-  Users  General discussion list for users.
+  Stack Overflow  http://stackoverflow.com/questions/tagged/cassandra;>Questions about 
using Cassandra.
+  
+  Users list  General mailing list for users.
   mailto:user-subscr...@cassandra.apache.org;>Subscribe
   | mailto:user-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/user@cassandra.apache.org/;>Archives
   
-  Developers  Development related discussion.
+  Developers list  Development related mailing list.
   mailto:dev-subscr...@cassandra.apache.org;>Subscribe
   | mailto:dev-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/dev@cassandra.apache.org/;>Archives
   
-  Commits  Commit notification source repository.
+  Commits list  Commit notifications.
   mailto:commits-subscr...@cassandra.apache.org;>Subscribe
   | mailto:commits-unsubscr...@cassandra.apache.org;>Unsubscribe
   | http://www.mail-archive.com/commits@cassandra.apache.org/;>Archives
-  Client Libraries  Discussion related to the development
-of idiomatic client APIs.
-  mailto:client-dev-subscr...@cassandra.apache.org;>Subscribe
-  | mailto:client-dev-unsubscr...@cassandra.apache.org;>Unsubscribe
-  | http://www.mail-archive.com/client-dev@cassandra.apache.org/;>Archives
   
 
   

[jira] [Updated] (CASSANDRA-11448) Running OOS should trigger the disk failure policy

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11448:

   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 2.1.x)
   (was: 3.x)
   3.5
   3.0.5
   2.2.6
   2.1.14
   Status: Resolved  (was: Ready to Commit)

[Committed|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=f3b3c410a0d84a4348cf05954b38df6b087762a7]
 the various versions.

On a second reading, I agree with you re: needing to catch on the singleton 
case, as it should just propagate up to {{ColumnFamilyStore.flushMemtable}} and 
be caught.

> Running OOS should trigger the disk failure policy
> --
>
> Key: CASSANDRA-11448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11448
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Brandon Williams
>Assignee: Branimir Lambov
> Fix For: 2.1.14, 2.2.6, 3.0.5, 3.5
>
>
> Currently when you run OOS, this happens:
> {noformat}
> ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047  
> CassandraDaemon.java:229 - Exception in thread 
> Thread[MemtableFlushWriter:8561,5,main]   java.lang.RuntimeException: 
> Insufficient disk space to write 48 bytes 
> at 
> org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29)
>  ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) 
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>  ~[guava-16.0.1.jar:na]
> at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120)
>  ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_66]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[na:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> {noformat}
> Now your flush writer is dead and postflush tasks build up forever.  Instead 
> we should throw FSWE and trigger the failure policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221878#comment-15221878
 ] 

Sylvain Lebresne commented on CASSANDRA-11349:
--

I suspect this doesn't affect 3.x: has someone checked, and if not, can someone 
do so we know if some 3.x version is needed or not for this?

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11448) Running OOS should trigger the disk failure policy

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11448:

Status: Ready to Commit  (was: Patch Available)

> Running OOS should trigger the disk failure policy
> --
>
> Key: CASSANDRA-11448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11448
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Brandon Williams
>Assignee: Branimir Lambov
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> Currently when you run OOS, this happens:
> {noformat}
> ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047  
> CassandraDaemon.java:229 - Exception in thread 
> Thread[MemtableFlushWriter:8561,5,main]   java.lang.RuntimeException: 
> Insufficient disk space to write 48 bytes 
> at 
> org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29)
>  ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) 
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>  ~[guava-16.0.1.jar:na]
> at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120)
>  ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_66]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[na:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> {noformat}
> Now your flush writer is dead and postflush tasks build up forever.  Instead 
> we should throw FSWE and trigger the failure policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[11/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jmckenzie]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd4cab24
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd4cab24
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd4cab24

Branch: refs/heads/cassandra-3.5
Commit: bd4cab24bb9582d15d6b4cf251e7da21230f1460
Parents: be38081 caaa9fc
Author: Josh McKenzie 
Authored: Fri Apr 1 11:47:39 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:49:41 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/CHANGES.txt
--
diff --cc CHANGES.txt
index 7fc628e,78ea961..b9376bc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -75,8 -38,10 +75,9 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
 + * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
 - * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 -   many threads to use (CASSANDRA-11179)
 - * Backport CASSANDRA-10679 (CASSANDRA-9598)
++ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)
   * Avoid dropping message too quickly due to missing unit conversion 
(CASSANDRA-11302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/Directories.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
index 8a8d0e7,636c73b..2ee4eed
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
@@@ -34,8 -34,10 +34,9 @@@ import java.util.concurrent.LinkedBlock
  import java.util.concurrent.TimeUnit;
  import java.util.concurrent.atomic.AtomicLong;
  
+ import com.google.common.annotations.VisibleForTesting;
  import com.google.common.collect.Iterables;
  import com.google.common.util.concurrent.*;
 -
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
index 000,8304aff..1527b1e
mode 00,100644..100644
--- a/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
+++ b/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
@@@ -1,0 -1,157 +1,157 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ package org.apache.cassandra.cql3;
+ 
+ import static junit.framework.Assert.fail;
+ 
+ import java.io.IOError;
+ import java.util.UUID;
+ import java.util.concurrent.ExecutionException;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ 
+ import 

[02/15] cassandra git commit: Fix out-of-space error treatment in memtable flushing

[jmckenzie]

Fix out-of-space error treatment in memtable flushing

Patch by Branimir Lambov; reviewed by Joshua McKenzie for
CASSANDRA-11448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3b3c410
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3b3c410
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3b3c410

Branch: refs/heads/cassandra-2.2
Commit: f3b3c410a0d84a4348cf05954b38df6b087762a7
Parents: 105fbb3
Author: Branimir Lambov 
Authored: Wed Mar 30 16:29:55 2016 +0300
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:45:18 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index af2518c..50bc894 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.14
+ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Backport CASSANDRA-10859 (CASSANDRA-11415)
  * COPY FROM fails when importing blob (CASSANDRA-11375)
  * Backport CASSANDRA-10679 (CASSANDRA-9598)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d66d3a..edc3564 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -967,6 +967,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
 final ReplayPosition lastReplayPosition;
+volatile FSWriteError flushFailure = null;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
 {
@@ -1009,12 +1010,16 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 // must check lastReplayPosition != null because Flush may find 
that all memtables are clean
 // and so not set a lastReplayPosition
-if (lastReplayPosition != null)
+// If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
+if (lastReplayPosition != null && flushFailure == null)
 {
 CommitLog.instance.discardCompletedSegments(metadata.cfId, 
lastReplayPosition);
 }
 
 metric.pendingFlushes.dec();
+
+if (flushFailure != null)
+throw flushFailure;
 }
 }
 
@@ -1114,11 +1119,20 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 metric.memtableSwitchCount.inc();
 
-for (Memtable memtable : memtables)
+try
+{
+for (Memtable memtable : memtables)
+{
+// flush the memtable
+
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
+reclaim(memtable);
+}
+}
+catch (FSWriteError e)
 {
-// flush the memtable
-
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
-reclaim(memtable);
+JVMStabilityInspector.inspectThrowable(e);
+// If we weren't killed, try to continue work but do not allow 
CommitLog to be discarded.
+postFlush.flushFailure = e;
 }
 
 // signal the post-flush we've done our work

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index 810c336..35aa447 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -310,7 +310,7 @@ public class Directories
 if (tooBig)
 return null;
 else
-throw 

[01/15] cassandra git commit: Fix out-of-space error treatment in memtable flushing

[jmckenzie]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 105fbb35d -> f3b3c410a
  refs/heads/cassandra-2.2 106750c78 -> caaa9fc82
  refs/heads/cassandra-3.0 be3808140 -> bd4cab24b
  refs/heads/cassandra-3.5 4111e4fdf -> af9b9cd5a
  refs/heads/trunk bdae4520c -> 6d1254ac2


Fix out-of-space error treatment in memtable flushing

Patch by Branimir Lambov; reviewed by Joshua McKenzie for
CASSANDRA-11448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3b3c410
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3b3c410
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3b3c410

Branch: refs/heads/cassandra-2.1
Commit: f3b3c410a0d84a4348cf05954b38df6b087762a7
Parents: 105fbb3
Author: Branimir Lambov 
Authored: Wed Mar 30 16:29:55 2016 +0300
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:45:18 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index af2518c..50bc894 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.14
+ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Backport CASSANDRA-10859 (CASSANDRA-11415)
  * COPY FROM fails when importing blob (CASSANDRA-11375)
  * Backport CASSANDRA-10679 (CASSANDRA-9598)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d66d3a..edc3564 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -967,6 +967,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
 final ReplayPosition lastReplayPosition;
+volatile FSWriteError flushFailure = null;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
 {
@@ -1009,12 +1010,16 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 // must check lastReplayPosition != null because Flush may find 
that all memtables are clean
 // and so not set a lastReplayPosition
-if (lastReplayPosition != null)
+// If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
+if (lastReplayPosition != null && flushFailure == null)
 {
 CommitLog.instance.discardCompletedSegments(metadata.cfId, 
lastReplayPosition);
 }
 
 metric.pendingFlushes.dec();
+
+if (flushFailure != null)
+throw flushFailure;
 }
 }
 
@@ -1114,11 +1119,20 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 metric.memtableSwitchCount.inc();
 
-for (Memtable memtable : memtables)
+try
+{
+for (Memtable memtable : memtables)
+{
+// flush the memtable
+
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
+reclaim(memtable);
+}
+}
+catch (FSWriteError e)
 {
-// flush the memtable
-
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
-reclaim(memtable);
+JVMStabilityInspector.inspectThrowable(e);
+// If we weren't killed, try to continue work but do not allow 
CommitLog to be discarded.
+postFlush.flushFailure = e;
 }
 
 // signal the post-flush we've done our work

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 

[13/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[jmckenzie]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af9b9cd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af9b9cd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af9b9cd5

Branch: refs/heads/cassandra-3.5
Commit: af9b9cd5ae248746475ada5434d08370148bbce0
Parents: 4111e4f bd4cab2
Author: Josh McKenzie 
Authored: Fri Apr 1 11:51:13 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:52:45 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 134 ++--
 .../org/apache/cassandra/db/Directories.java|  16 +-
 src/java/org/apache/cassandra/db/Memtable.java  |  29 +++-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|  54 ---
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 7 files changed, 276 insertions(+), 119 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/af9b9cd5/CHANGES.txt
--
diff --cc CHANGES.txt
index b542a67,b9376bc..73d9ab5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -109,9 -75,9 +109,10 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
 + * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
   * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)
   * Avoid dropping message too quickly due to missing unit conversion 
(CASSANDRA-11302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/af9b9cd5/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index f193c4d,aa12b80..6e01b34
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -910,6 -893,7 +910,7 @@@ public class ColumnFamilyStore implemen
  final OpOrder.Barrier writeBarrier;
  final CountDownLatch latch = new CountDownLatch(1);
  final ReplayPosition lastReplayPosition;
 -volatile FSWriteError flushFailure = null;
++volatile Throwable flushFailure = null;
  
  private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
  {
@@@ -951,6 -936,9 +953,9 @@@
  }
  
  metric.pendingFlushes.dec();
+ 
+ if (flushFailure != null)
 -throw flushFailure;
++throw Throwables.propagate(flushFailure);
  }
  }
  
@@@ -1050,81 -1038,26 +1055,106 @@@
  
  metric.memtableSwitchCount.inc();
  
- for (Memtable memtable : memtables)
+ try
  {
- List futures = new ArrayList<>();
- long totalBytesOnDisk = 0;
- long maxBytesOnDisk = 0;
- long minBytesOnDisk = Long.MAX_VALUE;
- List sstables = new ArrayList<>();
- try (LifecycleTransaction txn = 
LifecycleTransaction.offline(OperationType.FLUSH))
+ for (Memtable memtable : memtables)
+ {
 -// flush the memtable
 -
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
 -reclaim(memtable);
++flushMemtable(memtable);
+ }
+ }
 -catch (FSWriteError e)
++catch (Throwable t)
+ {
 -JVMStabilityInspector.inspectThrowable(e);
 -// If we weren't killed, try to continue work but do not 
allow CommitLog to be discarded.
 -postFlush.flushFailure = e;
++JVMStabilityInspector.inspectThrowable(t);
++postFlush.flushFailure = t;
+ }
 -
+ // signal the post-flush we've done our work
+ postFlush.latch.countDown();
+ }
+ 
++public void flushMemtable(Memtable memtable)
++{
++List futures = new ArrayList<>();
++long totalBytesOnDisk = 0;
++long maxBytesOnDisk = 0;
++long minBytesOnDisk = 

[06/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jmckenzie]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/caaa9fc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/caaa9fc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/caaa9fc8

Branch: refs/heads/trunk
Commit: caaa9fc82cd51b7321da4dad0715ebc561d180dd
Parents: 106750c f3b3c41
Author: Josh McKenzie 
Authored: Fri Apr 1 11:46:15 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:46:15 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/CHANGES.txt
--
diff --cc CHANGES.txt
index 3e51935,50bc894..78ea961
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,45 -1,7 +1,46 @@@
 -2.1.14
 +2.2.6
 + * Add cassandra-stress keystore option (CASSANDRA-9325)
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
 - * Backport CASSANDRA-10859 (CASSANDRA-11415)
 - * COPY FROM fails when importing blob (CASSANDRA-11375)
 + * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 + * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
 + * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
 + * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
 + * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
 + * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
 + * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
 + * Fix bloom filter sizing with LCS (CASSANDRA-11344)
 + * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)
 + * Fix intra-node serialization issue for multicolumn-restrictions 
(CASSANDRA-11196)
 + * Non-obsoleting compaction operations over compressed files can impose rate 
limit on normal reads (CASSANDRA-11301)
 + * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 + * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 + * Fix filtering on non-primary key columns for thrift static column families
 +   (CASSANDRA-6377)
 + * Only log yaml config once, at startup (CASSANDRA-11217)
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 + * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
 + * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
 + * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)
 + * (cqlsh) cqlsh cannot be called through symlink (CASSANDRA-11037)
 + * fix ohc and java-driver pom dependencies in build.xml (CASSANDRA-10793)
 + * Protect from keyspace dropped during repair (CASSANDRA-11065)
 + * Handle adding fields to a UDT in SELECT JSON and toJson() (CASSANDRA-11146)
 + * Better error message for cleanup (CASSANDRA-10991)
 + * cqlsh pg-style-strings broken if line ends with ';' (CASSANDRA-11123)
 + * Use cloned TokenMetadata in size estimates to avoid race against 
membership check
 +   (CASSANDRA-10736)
 + * Always persist upsampled index summaries (CASSANDRA-10512)
 + * (cqlsh) Fix inconsistent auto-complete (CASSANDRA-10733)
 + * Make SELECT JSON and toJson() threadsafe (CASSANDRA-11048)
 + * Fix SELECT on tuple relations for mixed ASC/DESC clustering order 
(CASSANDRA-7281)
 + * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
 + * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 +   (CASSANDRA-10010)
 +Merged from 2.1:
 + * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 +   many threads to use (CASSANDRA-11179)
   * Backport CASSANDRA-10679 (CASSANDRA-9598)
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[09/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jmckenzie]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/caaa9fc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/caaa9fc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/caaa9fc8

Branch: refs/heads/cassandra-3.5
Commit: caaa9fc82cd51b7321da4dad0715ebc561d180dd
Parents: 106750c f3b3c41
Author: Josh McKenzie 
Authored: Fri Apr 1 11:46:15 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:46:15 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/CHANGES.txt
--
diff --cc CHANGES.txt
index 3e51935,50bc894..78ea961
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,45 -1,7 +1,46 @@@
 -2.1.14
 +2.2.6
 + * Add cassandra-stress keystore option (CASSANDRA-9325)
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
 - * Backport CASSANDRA-10859 (CASSANDRA-11415)
 - * COPY FROM fails when importing blob (CASSANDRA-11375)
 + * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 + * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
 + * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
 + * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
 + * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
 + * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
 + * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
 + * Fix bloom filter sizing with LCS (CASSANDRA-11344)
 + * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)
 + * Fix intra-node serialization issue for multicolumn-restrictions 
(CASSANDRA-11196)
 + * Non-obsoleting compaction operations over compressed files can impose rate 
limit on normal reads (CASSANDRA-11301)
 + * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 + * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 + * Fix filtering on non-primary key columns for thrift static column families
 +   (CASSANDRA-6377)
 + * Only log yaml config once, at startup (CASSANDRA-11217)
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 + * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
 + * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
 + * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)
 + * (cqlsh) cqlsh cannot be called through symlink (CASSANDRA-11037)
 + * fix ohc and java-driver pom dependencies in build.xml (CASSANDRA-10793)
 + * Protect from keyspace dropped during repair (CASSANDRA-11065)
 + * Handle adding fields to a UDT in SELECT JSON and toJson() (CASSANDRA-11146)
 + * Better error message for cleanup (CASSANDRA-10991)
 + * cqlsh pg-style-strings broken if line ends with ';' (CASSANDRA-11123)
 + * Use cloned TokenMetadata in size estimates to avoid race against 
membership check
 +   (CASSANDRA-10736)
 + * Always persist upsampled index summaries (CASSANDRA-10512)
 + * (cqlsh) Fix inconsistent auto-complete (CASSANDRA-10733)
 + * Make SELECT JSON and toJson() threadsafe (CASSANDRA-11048)
 + * Fix SELECT on tuple relations for mixed ASC/DESC clustering order 
(CASSANDRA-7281)
 + * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
 + * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 +   (CASSANDRA-10010)
 +Merged from 2.1:
 + * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 +   many threads to use (CASSANDRA-11179)
   * Backport CASSANDRA-10679 (CASSANDRA-9598)
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[03/15] cassandra git commit: Fix out-of-space error treatment in memtable flushing

[jmckenzie]

Fix out-of-space error treatment in memtable flushing

Patch by Branimir Lambov; reviewed by Joshua McKenzie for
CASSANDRA-11448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3b3c410
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3b3c410
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3b3c410

Branch: refs/heads/trunk
Commit: f3b3c410a0d84a4348cf05954b38df6b087762a7
Parents: 105fbb3
Author: Branimir Lambov 
Authored: Wed Mar 30 16:29:55 2016 +0300
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:45:18 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index af2518c..50bc894 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.14
+ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Backport CASSANDRA-10859 (CASSANDRA-11415)
  * COPY FROM fails when importing blob (CASSANDRA-11375)
  * Backport CASSANDRA-10679 (CASSANDRA-9598)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d66d3a..edc3564 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -967,6 +967,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
 final ReplayPosition lastReplayPosition;
+volatile FSWriteError flushFailure = null;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
 {
@@ -1009,12 +1010,16 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 // must check lastReplayPosition != null because Flush may find 
that all memtables are clean
 // and so not set a lastReplayPosition
-if (lastReplayPosition != null)
+// If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
+if (lastReplayPosition != null && flushFailure == null)
 {
 CommitLog.instance.discardCompletedSegments(metadata.cfId, 
lastReplayPosition);
 }
 
 metric.pendingFlushes.dec();
+
+if (flushFailure != null)
+throw flushFailure;
 }
 }
 
@@ -1114,11 +1119,20 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 metric.memtableSwitchCount.inc();
 
-for (Memtable memtable : memtables)
+try
+{
+for (Memtable memtable : memtables)
+{
+// flush the memtable
+
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
+reclaim(memtable);
+}
+}
+catch (FSWriteError e)
 {
-// flush the memtable
-
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
-reclaim(memtable);
+JVMStabilityInspector.inspectThrowable(e);
+// If we weren't killed, try to continue work but do not allow 
CommitLog to be discarded.
+postFlush.flushFailure = e;
 }
 
 // signal the post-flush we've done our work

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index 810c336..35aa447 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -310,7 +310,7 @@ public class Directories
 if (tooBig)
 return null;
 else
-throw new 

[15/15] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

[jmckenzie]

Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6d1254ac
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6d1254ac
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6d1254ac

Branch: refs/heads/trunk
Commit: 6d1254ac238604f5fe6c5660eb72d40959626651
Parents: bdae452 af9b9cd
Author: Josh McKenzie 
Authored: Fri Apr 1 11:54:51 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:54:51 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 134 ++--
 .../org/apache/cassandra/db/Directories.java|  16 +-
 src/java/org/apache/cassandra/db/Memtable.java  |  29 +++-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|  56 ---
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 7 files changed, 276 insertions(+), 121 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d1254ac/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d1254ac/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index a6645b5,6e01b34..b47cf85
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -1128,17 -1149,10 +1160,15 @@@ public class ColumnFamilyStore implemen
  }
  }
  }
- memtable.cfs.replaceFlushed(memtable, sstables);
- reclaim(memtable);
+ }
+ memtable.cfs.replaceFlushed(memtable, sstables);
+ reclaim(memtable);
 -logger.debug("Flushed to {} ({} sstables, {} bytes), biggest {} 
bytes, smallest {} bytes", sstables, sstables.size(), totalBytesOnDisk, 
maxBytesOnDisk, minBytesOnDisk);
 +logger.debug("Flushed to {} ({} sstables, {}), biggest {}, 
smallest {}",
 + sstables,
 + sstables.size(),
 + FBUtilities.prettyPrintMemory(totalBytesOnDisk),
 + FBUtilities.prettyPrintMemory(maxBytesOnDisk),
 + FBUtilities.prettyPrintMemory(minBytesOnDisk));
- }
- // signal the post-flush we've done our work
- postFlush.latch.countDown();
  }
  
  private void reclaim(final Memtable memtable)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d1254ac/src/java/org/apache/cassandra/db/Memtable.java
--

[05/15] cassandra git commit: Fix out-of-space error treatment in memtable flushing

[jmckenzie]

Fix out-of-space error treatment in memtable flushing

Patch by Branimir Lambov; reviewed by Joshua McKenzie for
CASSANDRA-11448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3b3c410
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3b3c410
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3b3c410

Branch: refs/heads/cassandra-3.5
Commit: f3b3c410a0d84a4348cf05954b38df6b087762a7
Parents: 105fbb3
Author: Branimir Lambov 
Authored: Wed Mar 30 16:29:55 2016 +0300
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:45:18 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index af2518c..50bc894 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.14
+ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Backport CASSANDRA-10859 (CASSANDRA-11415)
  * COPY FROM fails when importing blob (CASSANDRA-11375)
  * Backport CASSANDRA-10679 (CASSANDRA-9598)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d66d3a..edc3564 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -967,6 +967,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
 final ReplayPosition lastReplayPosition;
+volatile FSWriteError flushFailure = null;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
 {
@@ -1009,12 +1010,16 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 // must check lastReplayPosition != null because Flush may find 
that all memtables are clean
 // and so not set a lastReplayPosition
-if (lastReplayPosition != null)
+// If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
+if (lastReplayPosition != null && flushFailure == null)
 {
 CommitLog.instance.discardCompletedSegments(metadata.cfId, 
lastReplayPosition);
 }
 
 metric.pendingFlushes.dec();
+
+if (flushFailure != null)
+throw flushFailure;
 }
 }
 
@@ -1114,11 +1119,20 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 metric.memtableSwitchCount.inc();
 
-for (Memtable memtable : memtables)
+try
+{
+for (Memtable memtable : memtables)
+{
+// flush the memtable
+
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
+reclaim(memtable);
+}
+}
+catch (FSWriteError e)
 {
-// flush the memtable
-
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
-reclaim(memtable);
+JVMStabilityInspector.inspectThrowable(e);
+// If we weren't killed, try to continue work but do not allow 
CommitLog to be discarded.
+postFlush.flushFailure = e;
 }
 
 // signal the post-flush we've done our work

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index 810c336..35aa447 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -310,7 +310,7 @@ public class Directories
 if (tooBig)
 return null;
 else
-throw 

[12/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jmckenzie]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd4cab24
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd4cab24
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd4cab24

Branch: refs/heads/cassandra-3.0
Commit: bd4cab24bb9582d15d6b4cf251e7da21230f1460
Parents: be38081 caaa9fc
Author: Josh McKenzie 
Authored: Fri Apr 1 11:47:39 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:49:41 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/CHANGES.txt
--
diff --cc CHANGES.txt
index 7fc628e,78ea961..b9376bc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -75,8 -38,10 +75,9 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
 + * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
 - * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 -   many threads to use (CASSANDRA-11179)
 - * Backport CASSANDRA-10679 (CASSANDRA-9598)
++ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)
   * Avoid dropping message too quickly due to missing unit conversion 
(CASSANDRA-11302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/Directories.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
index 8a8d0e7,636c73b..2ee4eed
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
@@@ -34,8 -34,10 +34,9 @@@ import java.util.concurrent.LinkedBlock
  import java.util.concurrent.TimeUnit;
  import java.util.concurrent.atomic.AtomicLong;
  
+ import com.google.common.annotations.VisibleForTesting;
  import com.google.common.collect.Iterables;
  import com.google.common.util.concurrent.*;
 -
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
index 000,8304aff..1527b1e
mode 00,100644..100644
--- a/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
+++ b/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
@@@ -1,0 -1,157 +1,157 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ package org.apache.cassandra.cql3;
+ 
+ import static junit.framework.Assert.fail;
+ 
+ import java.io.IOError;
+ import java.util.UUID;
+ import java.util.concurrent.ExecutionException;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ 
+ import 

[14/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[jmckenzie]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af9b9cd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af9b9cd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af9b9cd5

Branch: refs/heads/trunk
Commit: af9b9cd5ae248746475ada5434d08370148bbce0
Parents: 4111e4f bd4cab2
Author: Josh McKenzie 
Authored: Fri Apr 1 11:51:13 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:52:45 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 134 ++--
 .../org/apache/cassandra/db/Directories.java|  16 +-
 src/java/org/apache/cassandra/db/Memtable.java  |  29 +++-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|  54 ---
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 7 files changed, 276 insertions(+), 119 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/af9b9cd5/CHANGES.txt
--
diff --cc CHANGES.txt
index b542a67,b9376bc..73d9ab5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -109,9 -75,9 +109,10 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
 + * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
   * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)
   * Avoid dropping message too quickly due to missing unit conversion 
(CASSANDRA-11302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/af9b9cd5/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index f193c4d,aa12b80..6e01b34
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -910,6 -893,7 +910,7 @@@ public class ColumnFamilyStore implemen
  final OpOrder.Barrier writeBarrier;
  final CountDownLatch latch = new CountDownLatch(1);
  final ReplayPosition lastReplayPosition;
 -volatile FSWriteError flushFailure = null;
++volatile Throwable flushFailure = null;
  
  private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
  {
@@@ -951,6 -936,9 +953,9 @@@
  }
  
  metric.pendingFlushes.dec();
+ 
+ if (flushFailure != null)
 -throw flushFailure;
++throw Throwables.propagate(flushFailure);
  }
  }
  
@@@ -1050,81 -1038,26 +1055,106 @@@
  
  metric.memtableSwitchCount.inc();
  
- for (Memtable memtable : memtables)
+ try
  {
- List futures = new ArrayList<>();
- long totalBytesOnDisk = 0;
- long maxBytesOnDisk = 0;
- long minBytesOnDisk = Long.MAX_VALUE;
- List sstables = new ArrayList<>();
- try (LifecycleTransaction txn = 
LifecycleTransaction.offline(OperationType.FLUSH))
+ for (Memtable memtable : memtables)
+ {
 -// flush the memtable
 -
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
 -reclaim(memtable);
++flushMemtable(memtable);
+ }
+ }
 -catch (FSWriteError e)
++catch (Throwable t)
+ {
 -JVMStabilityInspector.inspectThrowable(e);
 -// If we weren't killed, try to continue work but do not 
allow CommitLog to be discarded.
 -postFlush.flushFailure = e;
++JVMStabilityInspector.inspectThrowable(t);
++postFlush.flushFailure = t;
+ }
 -
+ // signal the post-flush we've done our work
+ postFlush.latch.countDown();
+ }
+ 
++public void flushMemtable(Memtable memtable)
++{
++List futures = new ArrayList<>();
++long totalBytesOnDisk = 0;
++long maxBytesOnDisk = 0;
++long minBytesOnDisk = Long.MAX_VALUE;

[07/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jmckenzie]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/caaa9fc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/caaa9fc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/caaa9fc8

Branch: refs/heads/cassandra-2.2
Commit: caaa9fc82cd51b7321da4dad0715ebc561d180dd
Parents: 106750c f3b3c41
Author: Josh McKenzie 
Authored: Fri Apr 1 11:46:15 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:46:15 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/CHANGES.txt
--
diff --cc CHANGES.txt
index 3e51935,50bc894..78ea961
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,45 -1,7 +1,46 @@@
 -2.1.14
 +2.2.6
 + * Add cassandra-stress keystore option (CASSANDRA-9325)
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
 - * Backport CASSANDRA-10859 (CASSANDRA-11415)
 - * COPY FROM fails when importing blob (CASSANDRA-11375)
 + * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 + * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
 + * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
 + * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
 + * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
 + * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
 + * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
 + * Fix bloom filter sizing with LCS (CASSANDRA-11344)
 + * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)
 + * Fix intra-node serialization issue for multicolumn-restrictions 
(CASSANDRA-11196)
 + * Non-obsoleting compaction operations over compressed files can impose rate 
limit on normal reads (CASSANDRA-11301)
 + * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 + * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 + * Fix filtering on non-primary key columns for thrift static column families
 +   (CASSANDRA-6377)
 + * Only log yaml config once, at startup (CASSANDRA-11217)
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 + * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
 + * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
 + * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)
 + * (cqlsh) cqlsh cannot be called through symlink (CASSANDRA-11037)
 + * fix ohc and java-driver pom dependencies in build.xml (CASSANDRA-10793)
 + * Protect from keyspace dropped during repair (CASSANDRA-11065)
 + * Handle adding fields to a UDT in SELECT JSON and toJson() (CASSANDRA-11146)
 + * Better error message for cleanup (CASSANDRA-10991)
 + * cqlsh pg-style-strings broken if line ends with ';' (CASSANDRA-11123)
 + * Use cloned TokenMetadata in size estimates to avoid race against 
membership check
 +   (CASSANDRA-10736)
 + * Always persist upsampled index summaries (CASSANDRA-10512)
 + * (cqlsh) Fix inconsistent auto-complete (CASSANDRA-10733)
 + * Make SELECT JSON and toJson() threadsafe (CASSANDRA-11048)
 + * Fix SELECT on tuple relations for mixed ASC/DESC clustering order 
(CASSANDRA-7281)
 + * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
 + * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 +   (CASSANDRA-10010)
 +Merged from 2.1:
 + * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 +   many threads to use (CASSANDRA-11179)
   * Backport CASSANDRA-10679 (CASSANDRA-9598)
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[08/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jmckenzie]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/caaa9fc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/caaa9fc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/caaa9fc8

Branch: refs/heads/cassandra-3.0
Commit: caaa9fc82cd51b7321da4dad0715ebc561d180dd
Parents: 106750c f3b3c41
Author: Josh McKenzie 
Authored: Fri Apr 1 11:46:15 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:46:15 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/CHANGES.txt
--
diff --cc CHANGES.txt
index 3e51935,50bc894..78ea961
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,45 -1,7 +1,46 @@@
 -2.1.14
 +2.2.6
 + * Add cassandra-stress keystore option (CASSANDRA-9325)
+  * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
 - * Backport CASSANDRA-10859 (CASSANDRA-11415)
 - * COPY FROM fails when importing blob (CASSANDRA-11375)
 + * Dont mark sstables as repairing with sub range repairs (CASSANDRA-11451)
 + * Fix use of NullUpdater for 2i during compaction (CASSANDRA-11450)
 + * Notify when sstables change after cancelling compaction (CASSANDRA-11373)
 + * cqlsh: COPY FROM should check that explicit column names are valid 
(CASSANDRA-11333)
 + * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
 + * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
 + * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
 + * Fix bloom filter sizing with LCS (CASSANDRA-11344)
 + * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)
 + * Fix intra-node serialization issue for multicolumn-restrictions 
(CASSANDRA-11196)
 + * Non-obsoleting compaction operations over compressed files can impose rate 
limit on normal reads (CASSANDRA-11301)
 + * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 + * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 + * Fix filtering on non-primary key columns for thrift static column families
 +   (CASSANDRA-6377)
 + * Only log yaml config once, at startup (CASSANDRA-11217)
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 + * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
 + * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
 + * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)
 + * (cqlsh) cqlsh cannot be called through symlink (CASSANDRA-11037)
 + * fix ohc and java-driver pom dependencies in build.xml (CASSANDRA-10793)
 + * Protect from keyspace dropped during repair (CASSANDRA-11065)
 + * Handle adding fields to a UDT in SELECT JSON and toJson() (CASSANDRA-11146)
 + * Better error message for cleanup (CASSANDRA-10991)
 + * cqlsh pg-style-strings broken if line ends with ';' (CASSANDRA-11123)
 + * Use cloned TokenMetadata in size estimates to avoid race against 
membership check
 +   (CASSANDRA-10736)
 + * Always persist upsampled index summaries (CASSANDRA-10512)
 + * (cqlsh) Fix inconsistent auto-complete (CASSANDRA-10733)
 + * Make SELECT JSON and toJson() threadsafe (CASSANDRA-11048)
 + * Fix SELECT on tuple relations for mixed ASC/DESC clustering order 
(CASSANDRA-7281)
 + * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
 + * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 +   (CASSANDRA-10010)
 +Merged from 2.1:
 + * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 +   many threads to use (CASSANDRA-11179)
   * Backport CASSANDRA-10679 (CASSANDRA-9598)
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/caaa9fc8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[10/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jmckenzie]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd4cab24
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd4cab24
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd4cab24

Branch: refs/heads/trunk
Commit: bd4cab24bb9582d15d6b4cf251e7da21230f1460
Parents: be38081 caaa9fc
Author: Josh McKenzie 
Authored: Fri Apr 1 11:47:39 2016 -0400
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:49:41 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/CHANGES.txt
--
diff --cc CHANGES.txt
index 7fc628e,78ea961..b9376bc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -75,8 -38,10 +75,9 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
 + * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
 - * Add a -j parameter to scrub/cleanup/upgradesstables to state how
 -   many threads to use (CASSANDRA-11179)
 - * Backport CASSANDRA-10679 (CASSANDRA-9598)
++ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
   * Don't do defragmentation if reading from repaired sstables 
(CASSANDRA-10342)
   * Fix streaming_socket_timeout_in_ms not enforced (CASSANDRA-11286)
   * Avoid dropping message too quickly due to missing unit conversion 
(CASSANDRA-11302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/Directories.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
--
diff --cc 
src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
index 8a8d0e7,636c73b..2ee4eed
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManager.java
@@@ -34,8 -34,10 +34,9 @@@ import java.util.concurrent.LinkedBlock
  import java.util.concurrent.TimeUnit;
  import java.util.concurrent.atomic.AtomicLong;
  
+ import com.google.common.annotations.VisibleForTesting;
  import com.google.common.collect.Iterables;
  import com.google.common.util.concurrent.*;
 -
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd4cab24/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
index 000,8304aff..1527b1e
mode 00,100644..100644
--- a/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
+++ b/test/unit/org/apache/cassandra/cql3/OutOfSpaceTest.java
@@@ -1,0 -1,157 +1,157 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ package org.apache.cassandra.cql3;
+ 
+ import static junit.framework.Assert.fail;
+ 
+ import java.io.IOError;
+ import java.util.UUID;
+ import java.util.concurrent.ExecutionException;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ 
+ import 

[04/15] cassandra git commit: Fix out-of-space error treatment in memtable flushing

[jmckenzie]

Fix out-of-space error treatment in memtable flushing

Patch by Branimir Lambov; reviewed by Joshua McKenzie for
CASSANDRA-11448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f3b3c410
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f3b3c410
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f3b3c410

Branch: refs/heads/cassandra-3.0
Commit: f3b3c410a0d84a4348cf05954b38df6b087762a7
Parents: 105fbb3
Author: Branimir Lambov 
Authored: Wed Mar 30 16:29:55 2016 +0300
Committer: Josh McKenzie 
Committed: Fri Apr 1 11:45:18 2016 -0400

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  24 ++-
 .../org/apache/cassandra/db/Directories.java|   2 +-
 .../db/commitlog/CommitLogSegmentManager.java   |   4 +-
 .../cassandra/io/util/DiskAwareRunnable.java|   5 +-
 .../apache/cassandra/cql3/OutOfSpaceTest.java   | 157 +++
 6 files changed, 185 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index af2518c..50bc894 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.14
+ * Fix out-of-space error treatment in memtable flushing (CASSANDRA-11448).
  * Backport CASSANDRA-10859 (CASSANDRA-11415)
  * COPY FROM fails when importing blob (CASSANDRA-11375)
  * Backport CASSANDRA-10679 (CASSANDRA-9598)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d66d3a..edc3564 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -967,6 +967,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
 final ReplayPosition lastReplayPosition;
+volatile FSWriteError flushFailure = null;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier, ReplayPosition lastReplayPosition)
 {
@@ -1009,12 +1010,16 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 // must check lastReplayPosition != null because Flush may find 
that all memtables are clean
 // and so not set a lastReplayPosition
-if (lastReplayPosition != null)
+// If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
+if (lastReplayPosition != null && flushFailure == null)
 {
 CommitLog.instance.discardCompletedSegments(metadata.cfId, 
lastReplayPosition);
 }
 
 metric.pendingFlushes.dec();
+
+if (flushFailure != null)
+throw flushFailure;
 }
 }
 
@@ -1114,11 +1119,20 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 metric.memtableSwitchCount.inc();
 
-for (Memtable memtable : memtables)
+try
+{
+for (Memtable memtable : memtables)
+{
+// flush the memtable
+
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
+reclaim(memtable);
+}
+}
+catch (FSWriteError e)
 {
-// flush the memtable
-
MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
-reclaim(memtable);
+JVMStabilityInspector.inspectThrowable(e);
+// If we weren't killed, try to continue work but do not allow 
CommitLog to be discarded.
+postFlush.flushFailure = e;
 }
 
 // signal the post-flush we've done our work

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f3b3c410/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index 810c336..35aa447 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -310,7 +310,7 @@ public class Directories
 if (tooBig)
 return null;
 else
-throw 

[jira] [Commented] (CASSANDRA-10805) Additional Compaction Logging

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221872#comment-15221872
 ] 

Carl Yeksigian commented on CASSANDRA-10805:


I've pushed a new version which addresses the comments. I started a 
{{CompactionLogger.Writer}} interface; wanted to know if this made sense, or if 
we should change to having these be objects that could be serialized as JSON to 
be serialized differently for different interfaces.

I've just kicked off new utests/dtests, so we'll see how it looks.

> Additional Compaction Logging
> -
>
> Key: CASSANDRA-10805
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10805
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction, Observability
>Reporter: Carl Yeksigian
>Assignee: Carl Yeksigian
>Priority: Minor
> Fix For: 3.x
>
>
> Currently, viewing the results of past compactions requires parsing the log 
> and looking at the compaction history system table, which doesn't have 
> information about, for example, flushed sstables not previously compacted.
> This is a proposal to extend the information captured for compaction. 
> Initially, this would be done through a JMX call, but if it proves to be 
> useful and not much overhead, it might be a feature that could be enabled for 
> the compaction strategy all the time.
> Initial log information would include:
> - The compaction strategy type controlling each column family
> - The set of sstables included in each compaction strategy
> - Information about flushes and compactions, including times and all involved 
> sstables
> - Information about sstables, including generation, size, and tokens
> - Any additional metadata the strategy wishes to add to a compaction or an 
> sstable, like the level of an sstable or the type of compaction being 
> performed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Fabien Rousseau (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221854#comment-15221854
 ] 

Fabien Rousseau commented on CASSANDRA-11349:
-

Thanks, the patch is OK.

In fact, the differences were produced by another bug that I will create 
separately.

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11477) MerkleTree mismatch when a cell is shadowed by a range tombstone in different SSTables

[Fabien Rousseau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabien Rousseau updated CASSANDRA-11477:

Attachment: 11477-2.1.patch

> MerkleTree mismatch when a cell is shadowed by a range tombstone in different 
> SSTables
> --
>
> Key: CASSANDRA-11477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11477
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Fabien Rousseau
>  Labels: repair
> Attachments: 11477-2.1.patch
>
>
> Below is a script which allows to reproduce the problem:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 text,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3) VALUES ( 'a', 'b', 'c');
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair test_rt table1
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> The wrong hash will be computed on node1, which will include the previously 
> deleted cell, thus resulting in a MT mismatch.
> This is due to the fact that, in LazilyCompactedRow, the RT is not added into 
> the RT tracker (in fact, it's added only if it is GCable, but should always 
> be added).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11477) MerkleTree mismatch when a cell is shadowed by a range tombstone in different SSTables

[Fabien Rousseau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabien Rousseau updated CASSANDRA-11477:

Reproduced In: 2.2.5, 2.1.13  (was: 2.1.13, 2.2.5)
   Status: Open  (was: Patch Available)

> MerkleTree mismatch when a cell is shadowed by a range tombstone in different 
> SSTables
> --
>
> Key: CASSANDRA-11477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11477
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Fabien Rousseau
>  Labels: repair
>
> Below is a script which allows to reproduce the problem:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 text,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3) VALUES ( 'a', 'b', 'c');
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair test_rt table1
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> The wrong hash will be computed on node1, which will include the previously 
> deleted cell, thus resulting in a MT mismatch.
> This is due to the fact that, in LazilyCompactedRow, the RT is not added into 
> the RT tracker (in fact, it's added only if it is GCable, but should always 
> be added).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11477) MerkleTree mismatch when a cell is shadowed by a range tombstone in different SSTables

[Fabien Rousseau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabien Rousseau updated CASSANDRA-11477:

Reproduced In: 2.2.5, 2.1.13  (was: 2.1.13, 2.2.5)
   Status: Patch Available  (was: Open)

> MerkleTree mismatch when a cell is shadowed by a range tombstone in different 
> SSTables
> --
>
> Key: CASSANDRA-11477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11477
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Fabien Rousseau
>  Labels: repair
>
> Below is a script which allows to reproduce the problem:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 text,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3) VALUES ( 'a', 'b', 'c');
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair test_rt table1
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> The wrong hash will be computed on node1, which will include the previously 
> deleted cell, thus resulting in a MT mismatch.
> This is due to the fact that, in LazilyCompactedRow, the RT is not added into 
> the RT tracker (in fact, it's added only if it is GCable, but should always 
> be added).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10943) NullPointer during LegacySchemaMigrator

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221863#comment-15221863
 ] 

Sylvain Lebresne commented on CASSANDRA-10943:
--

+1

> NullPointer during LegacySchemaMigrator
> ---
>
> Key: CASSANDRA-10943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10943
> Project: Cassandra
>  Issue Type: Bug
> Environment: Debian Jessie
> java version "1.8.0_66"
> Cassandra 2.2.4 -> 3.1.1 migration
>Reporter: Stephan Wienczny
>Assignee: Aleksey Yeschenko
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
> Attachments: 10943.txt
>
>
> I'm trying to upgrade my Cassandra cluster  from 2.2.4 to 3.1.1.
> I used
> {quote}
> nodetool upgradesstables
> nodetool drain
> {quote}
> before upgrading.  
> The result is this:
> {quote}
> INFO  [main] 2015-12-26 22:41:44,114 SystemKeyspace.java:1284 - Detected 
> version upgrade from 2.2.4 to 3.1.1, snapshotting system keyspace
> WARN  [main] 2015-12-26 22:41:44,318 CompressionParams.java:382 - The 
> sstable_compression option has been deprecated. You should use class instead
> ERROR [main] 2015-12-26 22:41:44,427 CassandraDaemon.java:690 - Exception 
> encountered during startup
> java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:33)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:24)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:114) 
> ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:272)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:264)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:250)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:221)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$218(LegacySchemaMigrator.java:214)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:214)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:163)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$215(LegacySchemaMigrator.java:154)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:154)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
>  ~[apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:223) 
> [apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:549)
>  [apache-cassandra-3.1.1.jar:3.1.1]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:677) 
> [apache-cassandra-3.1.1.jar:3.1.1]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11477) MerkleTree mismatch when a cell is shadowed by a range tombstone in different SSTables

[Fabien Rousseau (JIRA)]

Fabien Rousseau created CASSANDRA-11477:
---

 Summary: MerkleTree mismatch when a cell is shadowed by a range 
tombstone in different SSTables
 Key: CASSANDRA-11477
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11477
 Project: Cassandra
  Issue Type: Bug
Reporter: Fabien Rousseau
Assignee: Fabien Rousseau


Below is a script which allows to reproduce the problem:

{noformat}
ccm create test -v 2.1.13 -n 2 -s
ccm node1 cqlsh
CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': 2};
USE test_rt;
CREATE TABLE IF NOT EXISTS table1 (
c1 text,
c2 text,
c3 text,
PRIMARY KEY ((c1), c2)
);
INSERT INTO table1 (c1, c2, c3) VALUES ( 'a', 'b', 'c');
ctrl ^d
# now flush only one of the two nodes
ccm node1 flush 
ccm node1 cqlsh
USE test_rt;
DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
ctrl ^d
ccm node1 repair test_rt table1
# now grep the log and observe that there was some inconstencies detected 
between nodes (while it shouldn't have detected any)
ccm node1 showlog | grep "out of sync"
{noformat}

The wrong hash will be computed on node1, which will include the previously 
deleted cell, thus resulting in a MT mismatch.

This is due to the fact that, in LazilyCompactedRow, the RT is not added into 
the RT tracker (in fact, it's added only if it is GCable, but should always be 
added).





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10862) LCS repair: compact tables before making available in L0

[Jeff Ferland (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221843#comment-15221843
 ] 

Jeff Ferland commented on CASSANDRA-10862:
--

Note that CASSANDRA-10979 has helped immensely with the performance issues 
outlines here.

> LCS repair: compact tables before making available in L0
> 
>
> Key: CASSANDRA-10862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10862
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction, Streaming and Messaging
>Reporter: Jeff Ferland
>
> When doing repair on a system with lots of mismatched ranges, the number of 
> tables in L0 goes up dramatically, as correspondingly goes the number of 
> tables referenced for a query. Latency increases dramatically in tandem.
> Eventually all the copied tables are compacted down in L0, then copied into 
> L1 (which may be a very large copy), finally reducing the number of SSTables 
> per query into the manageable range.
> It seems to me that the cleanest answer is to compact after streaming, then 
> mark tables available rather than marking available when the file itself is 
> complete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11032) Full trace returned on ReadFailure by cqlsh

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11032:
-
Labels: cqlsh lhf  (was: cqlsh)

> Full trace returned on ReadFailure by cqlsh
> ---
>
> Key: CASSANDRA-11032
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11032
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Chris Splinter
>Priority: Minor
>  Labels: cqlsh, lhf
>
> I noticed that the full traceback is returned on a read failure where I 
> expected this to be a one line exception with the ReadFailure message. It is 
> minor, but would it be better to only return the ReadFailure details?
> {code}
> cqlsh> SELECT * FROM test_encryption_ks.test_bad_table;
> Traceback (most recent call last):
>   File "/usr/local/lib/dse/bin/../resources/cassandra/bin/cqlsh.py", line 
> 1276, in perform_simple_statement
> result = future.result()
>   File 
> "/usr/local/lib/dse/resources/cassandra/bin/../lib/cassandra-driver-internal-only-3.0.0-6af642d.zip/cassandra-driver-3.0.0-6af642d/cassandra/cluster.py",
>  line 3122, in result
> raise self._final_exception
> ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation 
> failed - received 0 responses and 1 failures" info={'failures': 1, 
> 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10998) Snapshots can fail during compactions

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-10998:
--
Issue Type: Bug  (was: Improvement)

> Snapshots can fail during compactions
> -
>
> Key: CASSANDRA-10998
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10998
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Minor
>
> In an unlucky, unlikely and rare situation, a snapshot can fail when sstables 
> are removed (due to compaction) while the snapshots being taken.
> So - for example:
> # Compaction starts (compact sstable 1, 2, 3 to 4)
> # Snapshot operation starts with sstables 1, 2, 3
> # Compaction removes sstables 1, 2, 3
> # Snapshot tries to hard link to sstables 1, 2, 3 - failure
> I think it's a very rare but still possible race condition manifesting in 
> {{SSTableReader#createLinks}}, which has been mitigated by CASSANDRA-7066 for 
> 3.0 (introduced a {{File.exists}} before the actual creation of the hard 
> link).
> The problem without CASSANDRA-7066 is, that the snapshot just fails. The 
> problem with the if-clause is that the snapshot is inconsistent as it may not 
> contain all sstable components or even not all sstables without a failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221817#comment-15221817
 ] 

Russ Hatch commented on CASSANDRA-11225:


That's really interesting, thanks [~Stefania]. I've kicked off a new run with 
your latest dtest code here: 
https://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/59/

> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
> 
>
> Key: CASSANDRA-11225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11225
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_novnode_dtest #209
> error: "AssertionError: Failed to read value from sufficient number of nodes, 
> required 2 but got 1 - [574, 2]"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11476) dtest failure in pushed_notifications_test.TestVariousNotifications.tombstone_failure_threshold_message_test

[Philip Thompson (JIRA)]

Philip Thompson created CASSANDRA-11476:
---

 Summary: dtest failure in 
pushed_notifications_test.TestVariousNotifications.tombstone_failure_threshold_message_test
 Key: CASSANDRA-11476
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11476
 Project: Cassandra
  Issue Type: Test
Reporter: Philip Thompson
Assignee: DS Test Eng


The test was unable to find the tombstone failure threshold error in the logs

example failure:

http://cassci.datastax.com/job/cassandra-3.0_dtest/635/testReport/pushed_notifications_test/TestVariousNotifications/tombstone_failure_threshold_message_test

Failed on CassCI build cassandra-3.0_dtest #635



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10920) Let systemd know if Cassandra is running

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221808#comment-15221808
 ] 

Sylvain Lebresne commented on CASSANDRA-10920:
--

Would you mind attaching your fix as a patch and marking this patch available?

> Let systemd know if Cassandra is running
> 
>
> Key: CASSANDRA-10920
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10920
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Packaging
>Reporter: Severin Leonhardt
>Priority: Minor
>  Labels: lhf
>
> Checking the status of the Cassandra service on CentOS 7 shows this:
> {noformat}
> $ systemctl status cassandra
> ● cassandra.service - SYSV: Starts and stops Cassandra
>Loaded: loaded (/etc/rc.d/init.d/cassandra)
>Active: active (exited) since Fri 2015-12-18 13:30:02 CET; 3 days ago
>  Docs: man:systemd-sysv-generator(8)
> ...
> {noformat}
> The same is shown after the Cassandra process is killed or dies by itself. 
> Notice this part: _Active: active (exited)_
> Starting Cassandra through {{systemctl start cassandra}} will not work, 
> because systemd still thinks the service is running.
> According to the [systemd documentation regarding compatibility with 
> SysV|http://www.freedesktop.org/wiki/Software/systemd/Incompatibilities/] 
> this is expected behavior. Systemd doesn't know the PID of the Cassandra 
> process and thus can't check if the service is running or not.
> It is suggested to add the following to the SysV script header comment:
> {noformat}
> # pidfile: /var/run/cassandra/cassandra.pid
> {noformat}
> After adding this manually systemd correctly detected the state of the 
> Cassandra process on our machines.
> *Update: /var/run/cassandra is missing after reboot*
> Since CentOS 7 /var/run is a symlink to /run which is tmpfs-mounted. This 
> makes /var/run/cassandra go away after a reboot and the PID file is never 
> created again. (With the above change systemd even fails because it can't 
> find the PID file.)
> I suggest adding this to make sure the directory for the PID file exists:
> {noformat}
> start)
> # Cassandra startup
> echo -n "Starting Cassandra: "
> +   pid_dir=`dirname $pid_file`
> +   [ -d $pid_dir ] || install -m 755 -o $CASSANDRA_OWNR -d $pid_dir
> su $CASSANDRA_OWNR -c "$CASSANDRA_PROG -p $pid_file" > $log_file 2>&1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)