[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144785#comment-16144785
 ] 

Hannu Kröger commented on CASSANDRA-13752:
--

[~jjirsa] We saw off by one in both directions. I will check with the guy who 
wrote a tool to fix the stats if he saw other values.

Also, might be that this doesn't fix the original problem causing it but at 
least it shouldn't write size different from amount of entries actually 
written. :P

Anyways, if there is some better approach, by all means.

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144774#comment-16144774
 ] 

Hannu Kröger commented on CASSANDRA-13752:
--

[~krummas] We are not sure if it was compaction or streaming. But the sizes are 
8-45GB and the typical size was around 20-25GB.

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13756) StreamingHistogram is not thread safe

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131453#comment-16131453
 ] 

Jeff Jirsa edited comment on CASSANDRA-13756 at 8/29/17 5:43 AM:
-

Shouldn't need a version for trunk, but [~jasobrown] if you can check me there 
to be sure that'd be nice (I think in the faster rewrite for trunk, we now 
[build|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java#L182-L186]
 a snapshot that is no longer modified on read).
 
|| branch || utest || dtest ||
| [3.0|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.0-13756] | [3.0 
circle|https://circleci.com/gh/jeffjirsa/cassandra/tree/cassandra-3.0-13756] | 
[3.0 
dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/221/]
 |
| [3.11|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.11-13756] | 
[3.11 
circle|https://circleci.com/gh/jeffjirsa/cassandra/tree/cassandra-3.11-13756] | 
[3.11 
dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/222/]
 |



was (Author: jjirsa):
Shouldn't need a version for trunk, but [~jasobrown] if you can check me there 
to be sure that'd be nice (I think in the faster rewrite for trunk, we now 
[build|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java#L182-L186]
 a snapshot that is no longer modified on read).
 
|| branch || utest || dtest ||
| [3.0|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.0-13756] | [3.0 
circle|https://circleci.com/gh/jeffjirsa/cassandra/tree/cassandra-3.0-13756] | 
[3.0 
dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/189/]
 |
| [3.11|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.11-13756] | 
[3.11 
circle|https://circleci.com/gh/jeffjirsa/cassandra/tree/cassandra-3.11-13756] | 
[3.11 
dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/190/]
 |


> StreamingHistogram is not thread safe
> -
>
> Key: CASSANDRA-13756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13756
> Project: Cassandra
>  Issue Type: Bug
>Reporter: xiangzhou xia
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x
>
>
> When we test C*3 in shadow cluster, we notice after a period of time, several 
> data node suddenly run into 100% cpu and stop process query anymore.
> After investigation, we found that threads are stuck on the sum() in 
> streaminghistogram class. Those are jmx threads that working on expose 
> getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is 
> a chance that multiple jmx thread is access streaminghistogram at the same 
> time).  
> After further investigation, we find that the optimization in CASSANDRA-13038 
> led to a spool flush every time when we call sum(). Since TreeMap is not 
> thread safe, threads will be stuck when multiple threads visit sum() at the 
> same time.
> There are two approaches to solve this issue. 
> The first one is to add a lock to the flush in sum() which will introduce 
> some extra overhead to streaminghistogram.
> The second one is to avoid streaminghistogram to be access by multiple 
> threads. For our specific case, is to remove the metrics we added.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13752:
---
Comment: was deleted

(was: [~hkroger] - I thought I remember reading (perhaps in IRC) that you 
opened this in a debugger and saw the corruption was that the length of the 
bins was off by one. Was the length field 1 less than the bins, or 1 greater 
than actual number of bins? 
)

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

[Malcolm Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144759#comment-16144759
 ] 

Malcolm Taylor commented on CASSANDRA-13799:


The issues are not all trivial, but each proposed change will be very small 
(typically one or two lines each).

> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144756#comment-16144756
 ] 

Jeff Jirsa commented on CASSANDRA-13752:


[~hkroger] - I thought I remember reading (perhaps in IRC) that you opened this 
in a debugger and saw the corruption was that the length of the bins was off by 
one. Was the length field 1 less than the bins, or 1 greater than actual number 
of bins? 


> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144743#comment-16144743
 ] 

Marcus Eriksson edited comment on CASSANDRA-13752 at 8/29/17 5:04 AM:
--

[~hkroger] how was the corrupted sstable written? 
(compaction/flush/anticompaction?)

edit: ok it is 16GB, so I guess it was either compaction or anticompaction
edit2: I guess it could be streamed as well


was (Author: krummas):
[~hkroger] how was the corrupted sstable written? 
(compaction/flush/anticompaction?)

edit: ok it is 16GB, so I guess it was either compaction or anticompaction

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144743#comment-16144743
 ] 

Marcus Eriksson edited comment on CASSANDRA-13752 at 8/29/17 5:01 AM:
--

[~hkroger] how was the corrupted sstable written? 
(compaction/flush/anticompaction?)

edit: ok it is 16GB, so I guess it was either compaction or anticompaction


was (Author: krummas):
[~hkroger] how was the corrupted sstable written? 
(compaction/flush/anticompaction?)

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144743#comment-16144743
 ] 

Marcus Eriksson commented on CASSANDRA-13752:
-

[~hkroger] how was the corrupted sstable written? 
(compaction/flush/anticompaction?)

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13798) Disallow filtering on non-primary-key base column for MV

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141491#comment-16141491
 ] 

ZhaoYang edited comment on CASSANDRA-13798 at 8/29/17 3:58 AM:
---

| [trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13798-trunk] 
|
| [3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13798-3.11] |

The patch is to revert supporting non-pk base column filtering on MV on 3.11 & 
trunk.  The feature was added in 3.10 and the consequence may have been 
overlooked. It's better to revert it in 3.x/trunk, before more users start 
using it.

At the same time, I will work on a proper design (eg. multi-timestamp approach 
discussed in CASSANDRA-11500 and CASSANDRA-10226) for non-pk base column 
filtering and multiple non-pk base column in view pk.

[~pauloricardomg] [~brstgt] [~KurtG] what do you think?


was (Author: jasonstack):
| [trunk|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13798-trunk] 
|
| [3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13798-3.11] |
| [dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-13798] |

The patch is to revert supporting non-pk base column filtering on MV on 3.11 & 
trunk.  The feature was added in 3.10 and the consequence may have been 
overlooked. It's better to revert it in 3.x/trunk, before more users start 
using it.

At the same time, I will work on a proper design (eg. multi-timestamp approach 
discussed in CASSANDRA-11500 and CASSANDRA-10226) for non-pk base column 
filtering and multiple non-pk base column in view pk.

[~pauloricardomg] [~brstgt] [~KurtG] what do you think?

> Disallow filtering on non-primary-key base column for MV
> 
>
> Key: CASSANDRA-13798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13798
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> We should probably consider disallow filtering conditions on non-primary-key 
> base column for Materialized View which is introduced in CASSANDRA-10368.
> The main problem is that the liveness of view row is now depending on 
> multiple base columns (multiple filtered non-pk base column + base column 
> used in view pk) and this semantic could not be properly support without 
> drastic storage format changes. (SEE CASSANDRA-11500, 
> [background|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16119823=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16119823])
> We should step back and re-consider the non-primary-key filtering feature 
> together with supporting multiple non-PK cols in MV clustering key in 
> CASSANDRA-10226.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13798) Disallow filtering on non-primary-key base column for MV

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144700#comment-16144700
 ] 

ZhaoYang commented on CASSANDRA-13798:
--

Updated NEWS.txt and introduced a flag: 
{{cassandra.mv.filtering.nonkey.columns=false}}.

If users "know what they are doing" ,eg. append-only, they could turn on the 
flag to create MV with non-pk column filtering.


> Disallow filtering on non-primary-key base column for MV
> 
>
> Key: CASSANDRA-13798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13798
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> We should probably consider disallow filtering conditions on non-primary-key 
> base column for Materialized View which is introduced in CASSANDRA-10368.
> The main problem is that the liveness of view row is now depending on 
> multiple base columns (multiple filtered non-pk base column + base column 
> used in view pk) and this semantic could not be properly support without 
> drastic storage format changes. (SEE CASSANDRA-11500, 
> [background|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16119823=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16119823])
> We should step back and re-consider the non-primary-key filtering feature 
> together with supporting multiple non-PK cols in MV clustering key in 
> CASSANDRA-10226.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13694) sstabledump does not show full precision of timestamp columns

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144048#comment-16144048
 ] 

ZhaoYang edited comment on CASSANDRA-13694 at 8/29/17 3:32 AM:
---

[~varuna] the problem  here is that: type.toString() is not meant for outputing 
readable data, see CASSANDRA-13573.  I think it's better to use: 
{{json.writeRaw(type.toJSONString())}}.

Could you fix it for 3.11/trunk? maybe 3.0 as well



was (Author: jasonstack):
[~varuna] the problem  here is that: type.toString() is not meant for outputing 
readable data, see CASSANDRA-13573.  I think it's better to use: 
{{json.writeRaw(type.toJSONString())}}.

Could you fix it for 3.11/trunk?


> sstabledump does not show full precision of timestamp columns
> -
>
> Key: CASSANDRA-13694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13694
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Ubuntu 16.04 LTS
>Reporter: Tim Reeves
>Assignee: Varun Barala
>  Labels: patch-available
> Fix For: 3.7
>
> Attachments: CASSANDRA-13694-after-review.patch, CASSANDRA-13694.patch
>
>
> Create a table:
> CREATE TABLE test_table (
> unit_no bigint,
> event_code text,
> active_time timestamp,
> ack_time timestamp,
> PRIMARY KEY ((unit_no, event_code), active_time)
> ) WITH CLUSTERING ORDER BY (active_time DESC)
> Insert a row:
> INSERT INTO test_table (unit_no, event_code, active_time, ack_time)
>   VALUES (1234, 'TEST EVENT', toTimestamp(now()), 
> toTimestamp(now()));
> Verify that it is in the database with a full timestamp:
> cqlsh:pentaho> select * from test_table;
>  unit_no | event_code | active_time | ack_time
> -++-+-
> 1234 | TEST EVENT | 2017-07-14 14:52:39.919000+ | 2017-07-14 
> 14:52:39.919000+
> (1 rows)
> Write file:
> nodetool flush
> nodetool compact pentaho
> Use sstabledump:
> treeves@ubuntu:~$ sstabledump 
> /var/lib/cassandra/data/pentaho/test_table-99ba228068a311e7ac30953b79ac2c3e/mb-2-big-Data.db
> [
>   {
> "partition" : {
>   "key" : [ "1234", "TEST EVENT" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 38,
> "clustering" : [ "2017-07-14 15:52+0100" ],
> "liveness_info" : { "tstamp" : "2017-07-14T14:52:39.888701Z" },
> "cells" : [
>   { "name" : "ack_time", "value" : "2017-07-14 15:52+0100" }
> ]
>   }
> ]
>   }
> ]
> treeves@ubuntu:~$ 
> The timestamp in the cluster key, and the regular column, are both truncated 
> to the minute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang edited comment on CASSANDRA-12783 at 8/29/17 2:47 AM:
---

I think this design is the right direction to solve the OOM issue with MV.

1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 

bq. Not really sure that it will prevent OOMs, but description isn't really 
clear what that is about.

I think if we could achieve following 2 points, we could say base mutation 
causing OOM is resolved

2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one view mutation could exceed 
max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}

We may need some iterator mechansim to split a huge mutation into small chunks, 
as long as new batchlog could provide atomicity guarantee.


was (Author: jasonstack):
I think this design is the right direction to solve the OOM issue with MV.

1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 

bq. Not really sure that it will prevent OOMs, but description isn't really 
clear what that is about.

I think if we could achieve following 2 points, we could say base mutation 
causing OOM is resolved

2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one mutation could exceed max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang edited comment on CASSANDRA-12783 at 8/29/17 2:47 AM:
---

I think this design is the right direction to solve the OOM issue with MV.

1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 

bq. Not really sure that it will prevent OOMs, but description isn't really 
clear what that is about.

I think if we could achieve following 2 points, we could say base mutation 
causing OOM is resolved

2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates of the same base partition in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one view mutation could exceed 
max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}

We may need some iterator mechansim to split a huge mutation into small chunks, 
as long as new batchlog could provide atomicity guarantee.


was (Author: jasonstack):
I think this design is the right direction to solve the OOM issue with MV.

1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 

bq. Not really sure that it will prevent OOMs, but description isn't really 
clear what that is about.

I think if we could achieve following 2 points, we could say base mutation 
causing OOM is resolved

2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one view mutation could exceed 
max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}

We may need some iterator mechansim to split a huge mutation into small chunks, 
as long as new batchlog could provide atomicity guarantee.

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang edited comment on CASSANDRA-12783 at 8/29/17 2:44 AM:
---

I think this design is the right direction to solve the OOM issue with MV.

1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 

bq. Not really sure that it will prevent OOMs, but description isn't really 
clear what that is about.

I think if we could achieve following 2 points, we could say base mutation 
causing OOM is resolved

2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one mutation could exceed max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}


was (Author: jasonstack):
1. 
The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 


2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one mutation could exceed max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13780) ADD Node streaming throughput performance

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144667#comment-16144667
 ] 

Jeff Jirsa commented on CASSANDRA-13780:


{quote}
2. compaction on the newly added node - compaction has fallen behind, with 
anywhere from 4,000 to 10,000 SSTables at any given time. It took 3 weeks for 
compaction to finish on each newly added node. Increasing number of compaction 
threads to match number of CPU (40) and increasing compaction throughput to 
32MB/s seemed to be the sweet spot.
{quote}

This is a known limitation of vnodes, and has been since they were introduced 
in 1.2. It's less awful if you use LCS, but obviously that typically requires 
quite a bit more IO. I'm not going to link it here, but if you google "real 
world dtcs for operators", I cover this for time-series use cases in my 2015 
cassandra summit talk (which is where I introduced TWCS to solve problems I 
encountered doing exactly what you're trying to do now.

{quote}
3. TWCS buckets on new node, data streamed to this node over 4 1/2 days. 
Compaction correctly placed the data in daily files, but the problem is the 
file dates reflect when compaction created the file and not the date of the 
last record written in the TWCS bucket, which will cause the files to remain 
around much longer than necessary.
{quote}

TWCS ignores the file dates, it uses the sstable's max timestamp value from the 
metadata. This should be respected after streaming.

{quote}
1. What can be done to substantially improve the performance of adding a new 
node?
{quote}

Jason has been working on fixing streaming. Right now it's CPU bound rebuilding 
sstable components (which is CPU intensive, generates a lot of garbage 
re-compressing and whatnot).

{quote}
2. Can compaction on TWCS partitions for newly added nodes change the file 
create date to match the highest date record in the file or add another piece 
of meta-data to the TWCS files that reflect the file drop date so that TWCS 
partitions can be dropped consistently?
{quote}

Pretty sure this is a nonissue, because as I mentioned, TWCS doesnt care about 
the sstable's file creation date, it uses the metadata max timestamp.

{quote}
Requirement is to double cluster size, capacity, and ingestion volume within a 
few weeks.
{quote}

Usually people who need to grow rapidly tend not to use vnodes, so they can 
bootstrap more than one node at a time simultaneously. It's too late to give 
you that advice, though, so here's something slightly different:

If you can spare the hardware (easier in a cloud environment than physical), 
you can add a whole new "datacenter" with RF=0, then ALTER KEYSPACE to set RF=3 
(or whatever you use), then {{nodetool rebuild}} it in place to stream data to 
it all at once, then tear down the old one. That requires a bit of extra 
hardware, but let's you greatly increase cluster size very quickly.




> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
>  total   used   free sharedbuffers cached
> Mem:  252G   217G34G   708K   308M   149G
> -/+ buffers/cache:67G   185G
> Swap:  16G 0B16G
>Reporter: Kevin Rivait
> Fix For: 3.0.9
>
>
> Problem: Adding a new node to a large cluster runs at least 1000x slower than 
> what the network and node hardware capacity can support, taking several days 
> per new node.  Adjusting stream throughput and other YAML parameters seems to 
> have no effect on performance.  Essentially, it appears that Cassandra has an 
> architecture scalability growth problem when adding new nodes to a moderate 
> to high data ingestion cluster 

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144660#comment-16144660
 ] 

Kurt Greaves commented on CASSANDRA-13418:
--

>Watchers, feel free to jump on the question
May be missing something obvious but can someone clarify why you wouldn't want 
to enable {{uncheckedTombstoneCompaction}} when 
{{unsafe_aggressive_sstable_expiration}} is also enabled? 
What exactly are the implications of:
1. enabling both
2. only enabling unsafe_aggressive_sstable_expiration

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13780) ADD Node streaming throughput performance

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144649#comment-16144649
 ] 

Kurt Greaves commented on CASSANDRA-13780:
--

Unfortunately I don't have time to bury my head in the streaming code. Maybe 
someone with a better understanding can explain.

> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
>  total   used   free sharedbuffers cached
> Mem:  252G   217G34G   708K   308M   149G
> -/+ buffers/cache:67G   185G
> Swap:  16G 0B16G
>Reporter: Kevin Rivait
> Fix For: 3.0.9
>
>
> Problem: Adding a new node to a large cluster runs at least 1000x slower than 
> what the network and node hardware capacity can support, taking several days 
> per new node.  Adjusting stream throughput and other YAML parameters seems to 
> have no effect on performance.  Essentially, it appears that Cassandra has an 
> architecture scalability growth problem when adding new nodes to a moderate 
> to high data ingestion cluster because Cassandra cannot add new node capacity 
> fast enough to keep up with increasing data ingestion volumes and growth.
> Initial Configuration: 
> Running 3.0.9 and have implemented TWCS on one of our largest table.
> Largest table partitioned on (ID, MM)  using 1 day buckets with a TTL of 
> 60 days.
> Next release will change partitioning to (ID, MMDD) so that partitions 
> are aligned with daily TWCS buckets.
> Each node is currently creating roughly a 30GB SSTable per day.
> TWCS working as expected,  daily SSTables are dropping off daily after 70 
> days ( 60 + 10 day grace)
> Current deployment is a 28 node 2 datacenter cluster, 14 nodes in each DC , 
> replication factor 3
> Data directories are backed with 4 - 2TB SSDs on each node  and a 1 800GB SSD 
> for commit logs.
> Requirement is to double cluster size, capacity, and ingestion volume within 
> a few weeks.
> Observed Behavior:
> 1. streaming throughput during add node – we observed maximum 6 Mb/s 
> streaming from each of the 14 nodes on a 20Gb/s switched network, taking at 
> least 106 hours for each node to join cluster and each node is only about 2.2 
> TB is size.
> 2. compaction on the newly added node - compaction has fallen behind, with 
> anywhere from 4,000 to 10,000 SSTables at any given time.  It took 3 weeks 
> for compaction to finish on each newly added node.   Increasing number of 
> compaction threads to match number of CPU (40)  and increasing compaction 
> throughput to 32MB/s seemed to be the sweet spot. 
> 3. TWCS buckets on new node, data streamed to this node over 4 1/2 days.  
> Compaction correctly placed the data in daily files, but the problem is the 
> file dates reflect when compaction created the file and not the date of the 
> last record written in the TWCS bucket, which will cause the files to remain 
> around much longer than necessary.  
> Two Questions:
> 1. What can be done to substantially improve the performance of adding a new 
> node?
> 2. Can compaction on TWCS partitions for newly added nodes change the file 
> create date to match the highest date record in the file -or- add another 
> piece of meta-data to the TWCS files that reflect the file drop date so that 
> TWCS partitions can be dropped consistently?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13798) Disallow filtering on non-primary-key base column for MV

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144644#comment-16144644
 ] 

Kurt Greaves commented on CASSANDRA-13798:
--

Right OK. If we can do it in 4.1 that's fine. It seems we don't actually have a 
difference between major and minor then? My (incorrect) assumption was that 
massive storage changes constituted a major version change, and 4.x would be 
intended to be minor.




> Disallow filtering on non-primary-key base column for MV
> 
>
> Key: CASSANDRA-13798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13798
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> We should probably consider disallow filtering conditions on non-primary-key 
> base column for Materialized View which is introduced in CASSANDRA-10368.
> The main problem is that the liveness of view row is now depending on 
> multiple base columns (multiple filtered non-pk base column + base column 
> used in view pk) and this semantic could not be properly support without 
> drastic storage format changes. (SEE CASSANDRA-11500, 
> [background|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16119823=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16119823])
> We should step back and re-consider the non-primary-key filtering feature 
> together with supporting multiple non-PK cols in MV clustering key in 
> CASSANDRA-10226.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13782) Cassandra RPM has wrong owner for /usr/share directories

[Murukesh Mohanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Murukesh Mohanan updated CASSANDRA-13782:
-
Issue Type: Sub-task  (was: Bug)
Parent: CASSANDRA-13433

> Cassandra RPM has wrong owner for /usr/share directories
> 
>
> Key: CASSANDRA-13782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13782
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Packaging
>Reporter: Hannu Kröger
>  Labels: lhf
>
> Some Cassandra RPM directories are owned by cassandra user against the fedora 
> package guidelines.
> Offending lines: 
> https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136
> "Permissions on files MUST be set properly. Inside of /usr, files should be 
> owned by root:root unless a more specific user or group is needed for 
> security."
> - 
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144467#comment-16144467
 ] 

Jeff Jirsa commented on CASSANDRA-13752:


[~jasobrown] and I have been chatting about this today. We're looking for a 
solution. I don't think your patch actually solves this problem, and it may 
have existed before 13038 (though 13038 may have made it more likely).


> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-7544) Allow storage port to be configurable per node

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-7544:
--
Status: Open  (was: Patch Available)

> Allow storage port to be configurable per node
> --
>
> Key: CASSANDRA-7544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7544
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sam Overton
>Assignee: Ariel Weisberg
> Fix For: 4.x
>
>
> Currently storage_port must be configured identically on all nodes in a 
> cluster and it is assumed that this is the case when connecting to a remote 
> node.
> This prevents running in any environment that requires multiple nodes to be 
> able to bind to the same network interface, such as with many automatic 
> provisioning/deployment frameworks.
> The current solutions seems to be
> * use a separate network interface for each node deployed to the same box. 
> This puts a big requirement on IP allocation at large scale.
> * allow multiple clusters to be provisioned from the same resource pool, but 
> restrict allocation to a maximum of one node per host from each cluster, 
> assuming each cluster is running on a different storage port.
> It would make operations much simpler in these kind of environments if the 
> environment provisioning the resources could assign the ports to be used when 
> bringing up a new node on shared hardware.
> The changes required would be at least the following:
> 1. configure seeds as IP:port instead of just IP
> 2. gossip the storage port as part of a node's ApplicationState
> 3. refer internally to nodes by hostID instead of IP, since there will be 
> multiple nodes with the same IP
> (1) & (2) are mostly trivial and I already have a patch for these. The bulk 
> of the work to enable this is (3), and I would structure this as a separate 
> pre-requisite patch. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-5406) Allow CQL3 queries to do extra filter after getting the column slice on a composite primary key

[Jon Haddad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Haddad resolved CASSANDRA-5406.
---
Resolution: Duplicate

This has worked for a while now with ALLOW FILTERING and I believe is the 
result of CASSANDRA-6377.  Closing out as a duplicate.

> Allow CQL3 queries to do extra filter after getting the column slice on a 
> composite primary key
> ---
>
> Key: CASSANDRA-5406
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5406
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jeremiah Jordan
>Priority: Minor
>
> Let the following work:
> {noformat}
> CREATE TABLE "ChequeDeDup2" (
>   "bucketId" int,
>   "transitAba" int,
>   "transitAccount" bigint,
>   "serialNo" int,
>   amount bigint,
>   "subjectId" uuid,
>   "channelInd" ascii,
>   "creditAba" int,
>   "creditAccount" bigint,
>   "sourceTs" timestamp,
>   PRIMARY KEY ("bucketId", "transitAba", "transitAccount", "serialNo", 
> amount, "subjectId")
> )
> select * from "ChequeDeDup2" where "bucketId" = 198 and "transitAba" >= 101 
> and "transitAccount" = 198 and "serialNo" = 1 and "amount" = -1 order by 
> "transitAba" desc , "transitAccount" desc, "serialNo" desc, amount desc, 
> "subjectId" DESC limit 5;
> Bad Request: PRIMARY KEY part transitAccount cannot be restricted (preceding 
> part transitAba is either not restricted or by a non-EQ relation)
> {noformat}
> The assumption seems to be that it is better to serialize all the data with 
> transitAba >= 198 back to the client side.  But it is *less* efficient for 
> the server to serialize all this data back to the client than it is to 
> execute subsequent filters.
> The user should be allowed to trade off the CPU cost of filtering the data on 
> the server side with the IO cost of serializing all that data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
--
Attachment: 13740-3.0.15.txt

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
--
Attachment: (was: 13740-3.0.15.txt)

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Jaydeepkumar Chovatia (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144419#comment-16144419
 ] 

Jaydeepkumar Chovatia commented on CASSANDRA-13740:
---

Hi [~tuxslayer]

Isn’t the race condition is as following:

*Thread T1 - Time 1:* Removes {{HintStore}} by calling 
[HintsService.instance.excise 
|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L2268]
 but
at this time node has not yet been removed from [tokenMetadata  
|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L133]

*Thread T2 - Time 2:* Mutation stage does a lookup to [tokenMetadata  | 
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L2617]
 and finds node as valid hence it dumps hint for it.

*Thread T1 - Time 3:* Now removes node from [tokenMetadata  | 
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L2270]
 

Hence I decided it is safer to sleep for {{StorageService.RING_DELAY}} and then 
schedule optional {{removeOrphanHintFiles}}  task. Please let me know your 
comments.

I have also incorporated your review comment, please find updated patch 
attached as well as here: 
https://github.com/jaydeepkumar1984/cassandra/commit/16d4ab3316ab71a9a3b96ab67944384092be40ca

Jaydeep

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA

[jira] [Created] (CASSANDRA-13816) Document and test CAS and non-CAS batch behavior for deleting and inserting the same key

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-13816:
--

 Summary: Document and test CAS and non-CAS batch behavior for 
deleting and inserting the same key
 Key: CASSANDRA-13816
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13816
 Project: Cassandra
  Issue Type: Bug
  Components: Documentation and Website, Testing
Reporter: Ariel Weisberg
Priority: Minor
 Fix For: 3.x, 3.11.x, 4.x


Add/verify unit tests for inserting and deleting the same key for cell 
deletion, partition deletion, and range tombstone deletion for both CAS and 
non-CAS batches.

Don't change the existing behavior.

The behavior differs between batch and CAS so in the both the CAS and batch 
documentation mention that the behavior is not consistent. Make sure it is 
visible in both high level and reference docs for that functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13816) Document and test CAS and non-CAS batch behavior for deleting and inserting the same key

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg reassigned CASSANDRA-13816:
--

Assignee: Ariel Weisberg

> Document and test CAS and non-CAS batch behavior for deleting and inserting 
> the same key
> 
>
> Key: CASSANDRA-13816
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13816
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website, Testing
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Minor
> Fix For: 3.11.x, 4.x, 3.x
>
>
> Add/verify unit tests for inserting and deleting the same key for cell 
> deletion, partition deletion, and range tombstone deletion for both CAS and 
> non-CAS batches.
> Don't change the existing behavior.
> The behavior differs between batch and CAS so in the both the CAS and batch 
> documentation mention that the behavior is not consistent. Make sure it is 
> visible in both high level and reference docs for that functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-7544) Allow storage port to be configurable per node

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144374#comment-16144374
 ] 

Ariel Weisberg commented on CASSANDRA-7544:
---

I think that depends on the 4.0 release date. This was really kind of sort of 
ready to go in quite a while ago, but we decided to make it depend on Netty and 
using a single port for both SSL and non-SSL so that we would only use a single 
port throughout the code base rather than two and transition away from two 
ports later.

Jason says that CASSANDRA-10404 is almost done. I need to do a lot of rebasing. 
On to of Netty in C*, but also the Java and Python driver, and those haven't 
been reviewed yet.

> Allow storage port to be configurable per node
> --
>
> Key: CASSANDRA-7544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7544
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sam Overton
>Assignee: Ariel Weisberg
> Fix For: 4.x
>
>
> Currently storage_port must be configured identically on all nodes in a 
> cluster and it is assumed that this is the case when connecting to a remote 
> node.
> This prevents running in any environment that requires multiple nodes to be 
> able to bind to the same network interface, such as with many automatic 
> provisioning/deployment frameworks.
> The current solutions seems to be
> * use a separate network interface for each node deployed to the same box. 
> This puts a big requirement on IP allocation at large scale.
> * allow multiple clusters to be provisioned from the same resource pool, but 
> restrict allocation to a maximum of one node per host from each cluster, 
> assuming each cluster is running on a different storage port.
> It would make operations much simpler in these kind of environments if the 
> environment provisioning the resources could assign the ports to be used when 
> bringing up a new node on shared hardware.
> The changes required would be at least the following:
> 1. configure seeds as IP:port instead of just IP
> 2. gossip the storage port as part of a node's ApplicationState
> 3. refer internally to nodes by hostID instead of IP, since there will be 
> multiple nodes with the same IP
> (1) & (2) are mostly trivial and I already have a patch for these. The bulk 
> of the work to enable this is (3), and I would structure this as a separate 
> pre-requisite patch. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12696) Allow to change logging levels based on components

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12696:
---
Status: Ready to Commit  (was: Patch Available)

> Allow to change logging levels based on components
> --
>
> Key: CASSANDRA-12696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12696
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: lhf
> Attachments: 12696-trunk.patch
>
>
> Currently users are able to dynamically change logging configuration by using 
> {{nodetool setlogginglevel  }}. Unfortunately this requires to 
> know a bit about the Cassandra package hierarchy and gathering all the 
> involved packages/classes can be tedious, especially in troubleshooting 
> situations. What I'd like to have is a way to tell a user to "_when X 
> happens, enable debug logs for bootstrapping/repair/compactions/.._" by 
> simply running e.g. {{nodetool setlogginglevel bootstrap DEBUG}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13815) RPM package for client tools - cqlsh + nodetool

[Dennis (JIRA)]

Dennis created CASSANDRA-13815:
--

 Summary: RPM package for client tools - cqlsh + nodetool
 Key: CASSANDRA-13815
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13815
 Project: Cassandra
  Issue Type: Wish
  Components: Packaging
Reporter: Dennis
 Fix For: 3.11.x, 3.10


Feature request. 
I see you guys are picking up on the RPM packages. 
Thanks for that. That could even be improved if you could package the client 
tools as a separate  or client-only package as well. That package could hold 
cqlsh and nodetool for example. 
That would support centralized, automated backup or other maintenance 
processes. 

Now the admin is forced to login to the box in order to use these tools, which 
is not really best practice, security wise. The admin would need to know an ssh 
account as well as the cassandra admin account. 

So, benefits or usage of a client package (cqlsh+nodetool): 
# Supports automated maintenance scripts (simply yum the client tools to a 
temporary vm)
# Better security, as the admin doesn't need to ssh into the instance host.

Without having to pull the full Cassandra packages on the clients.
Datastax does have such client packages, but they don't support the community 
edition anymore, so I am hoping that you can do this going forward.

Thanks! 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-7544) Allow storage port to be configurable per node

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-7544:
--
Fix Version/s: (was: 3.11.x)
   4.x

> Allow storage port to be configurable per node
> --
>
> Key: CASSANDRA-7544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7544
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sam Overton
>Assignee: Ariel Weisberg
> Fix For: 4.x
>
>
> Currently storage_port must be configured identically on all nodes in a 
> cluster and it is assumed that this is the case when connecting to a remote 
> node.
> This prevents running in any environment that requires multiple nodes to be 
> able to bind to the same network interface, such as with many automatic 
> provisioning/deployment frameworks.
> The current solutions seems to be
> * use a separate network interface for each node deployed to the same box. 
> This puts a big requirement on IP allocation at large scale.
> * allow multiple clusters to be provisioned from the same resource pool, but 
> restrict allocation to a maximum of one node per host from each cluster, 
> assuming each cluster is running on a different storage port.
> It would make operations much simpler in these kind of environments if the 
> environment provisioning the resources could assign the ports to be used when 
> bringing up a new node on shared hardware.
> The changes required would be at least the following:
> 1. configure seeds as IP:port instead of just IP
> 2. gossip the storage port as part of a node's ApplicationState
> 3. refer internally to nodes by hostID instead of IP, since there will be 
> multiple nodes with the same IP
> (1) & (2) are mostly trivial and I already have a patch for these. The bulk 
> of the work to enable this is (3), and I would structure this as a separate 
> pre-requisite patch. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13752) Corrupted SSTables created in 3.11

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannu Kröger updated CASSANDRA-13752:
-
 Reviewer: Jeff Jirsa
Fix Version/s: 3.11.1
   Status: Patch Available  (was: In Progress)

> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Hannu Kröger
>Assignee: Hannu Kröger
>Priority: Blocker
> Fix For: 3.11.1
>
>
> We have discovered issues with corrupted SSTables. 
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - 
> Cannot read sstable 
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
>  Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, 
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug  7 08:37 
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra  10 Aug  7 08:37 
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug  7 08:37 
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra   75880 Aug  7 08:37 
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra   13762 Aug  7 08:37 
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra  882008 Aug  7 08:37 
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra  92 Aug  7 08:37 mc-6166-big-TOC.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-7544) Allow storage port to be configurable per node

[Jon Haddad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144187#comment-16144187
 ] 

Jon Haddad commented on CASSANDRA-7544:
---

Is this patch destined for 4.0?  

> Allow storage port to be configurable per node
> --
>
> Key: CASSANDRA-7544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7544
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sam Overton
>Assignee: Ariel Weisberg
> Fix For: 3.11.x
>
>
> Currently storage_port must be configured identically on all nodes in a 
> cluster and it is assumed that this is the case when connecting to a remote 
> node.
> This prevents running in any environment that requires multiple nodes to be 
> able to bind to the same network interface, such as with many automatic 
> provisioning/deployment frameworks.
> The current solutions seems to be
> * use a separate network interface for each node deployed to the same box. 
> This puts a big requirement on IP allocation at large scale.
> * allow multiple clusters to be provisioned from the same resource pool, but 
> restrict allocation to a maximum of one node per host from each cluster, 
> assuming each cluster is running on a different storage port.
> It would make operations much simpler in these kind of environments if the 
> environment provisioning the resources could assign the ports to be used when 
> bringing up a new node on shared hardware.
> The changes required would be at least the following:
> 1. configure seeds as IP:port instead of just IP
> 2. gossip the storage port as part of a node's ApplicationState
> 3. refer internally to nodes by hostID instead of IP, since there will be 
> multiple nodes with the same IP
> (1) & (2) are mostly trivial and I already have a patch for these. The bulk 
> of the work to enable this is (3), and I would structure this as a separate 
> pre-requisite patch. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8576) Primary Key Pushdown For Hadoop

[Jon Haddad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144184#comment-16144184
 ] 

Jon Haddad commented on CASSANDRA-8576:
---

Unfortunately this patch is pretty stale now as 2.x is no longer getting 
feature improvements.  Is there anything here that would be relevant for 4.0? 

> Primary Key Pushdown For Hadoop
> ---
>
> Key: CASSANDRA-8576
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8576
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Russell Spitzer
>Assignee: Alex Liu
> Fix For: 2.2.x
>
> Attachments: 8576-2.1-branch.txt, 8576-trunk.txt, 
> CASSANDRA-8576-v1-2.2-branch.txt, CASSANDRA-8576-v2-2.1-branch.txt, 
> CASSANDRA-8576-v3-2.1-branch.txt
>
>
> I've heard reports from several users that they would like to have predicate 
> pushdown functionality for hadoop (Hive in particular) based services. 
> Example usecase
> Table with wide partitions, one per customer
> Application team has HQL they would like to run on a single customer
> Currently time to complete scales with number of customers since Input Format 
> can't pushdown primary key predicate
> Current implementation requires a full table scan (since it can't recognize 
> that a single partition was specified)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang edited comment on CASSANDRA-12783 at 8/28/17 6:19 PM:
---

1. 
The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part.
You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code} 


2. When partition/range deletion happen, it'd be good to avoid holding entire 
view updates in memory and OOM.
   Maybe we could store part by part into new batchlog, throttle by 
max-mutation-size

3. Depends on the data modelling, one mutation could exceed max-mutation-size
{code} 
eg. base:   (pk, ck) v1 v2
view:   (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}


was (Author: jasonstack):
The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part, it compares MSB, then LSB

You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code}

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang edited comment on CASSANDRA-12783 at 8/28/17 5:59 PM:
---

The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part, it compares MSB, then LSB

You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code}


was (Author: jasonstack):
The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part, it compares MSB.

You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code}

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12783) Break up large MV mutations to prevent OOMs

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144131#comment-16144131
 ] 

ZhaoYang commented on CASSANDRA-12783:
--

The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first 
then machine part, it compares MSB.

You could try {{TimeUUIDType.compare}}

{code}
UUID V1, Most Significant Bit:
0x time_low
0x time_mid
0xF000 version
0x0FFF time_hi
{code}

> Break up large MV mutations to prevent OOMs
> ---
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
>Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder 
> because otherwise we would break the contract of the batchlog, where some 
> mutations may be written and pushed out before the whole batch log has been 
> saved.
> We would need to ensure that all of the updates make it to the batchlog 
> before allowing the batchlog manager to try to replay them, but also before 
> we start pushing out updates to the paired replicas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13694) sstabledump does not show full precision of timestamp columns

[ZhaoYang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang reassigned CASSANDRA-13694:


Assignee: Varun Barala
Reviewer: ZhaoYang

[~varuna] the problem  here is that: type.toString() is not meant for outputing 
readable data, see CASSANDRA-13573.  I think it's better to use: 
{{json.writeRaw(type.toJSONString())}}.

Could you fix it for 3.11/trunk?


> sstabledump does not show full precision of timestamp columns
> -
>
> Key: CASSANDRA-13694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13694
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Ubuntu 16.04 LTS
>Reporter: Tim Reeves
>Assignee: Varun Barala
>  Labels: patch-available
> Fix For: 3.7
>
> Attachments: CASSANDRA-13694-after-review.patch, CASSANDRA-13694.patch
>
>
> Create a table:
> CREATE TABLE test_table (
> unit_no bigint,
> event_code text,
> active_time timestamp,
> ack_time timestamp,
> PRIMARY KEY ((unit_no, event_code), active_time)
> ) WITH CLUSTERING ORDER BY (active_time DESC)
> Insert a row:
> INSERT INTO test_table (unit_no, event_code, active_time, ack_time)
>   VALUES (1234, 'TEST EVENT', toTimestamp(now()), 
> toTimestamp(now()));
> Verify that it is in the database with a full timestamp:
> cqlsh:pentaho> select * from test_table;
>  unit_no | event_code | active_time | ack_time
> -++-+-
> 1234 | TEST EVENT | 2017-07-14 14:52:39.919000+ | 2017-07-14 
> 14:52:39.919000+
> (1 rows)
> Write file:
> nodetool flush
> nodetool compact pentaho
> Use sstabledump:
> treeves@ubuntu:~$ sstabledump 
> /var/lib/cassandra/data/pentaho/test_table-99ba228068a311e7ac30953b79ac2c3e/mb-2-big-Data.db
> [
>   {
> "partition" : {
>   "key" : [ "1234", "TEST EVENT" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 38,
> "clustering" : [ "2017-07-14 15:52+0100" ],
> "liveness_info" : { "tstamp" : "2017-07-14T14:52:39.888701Z" },
> "cells" : [
>   { "name" : "ack_time", "value" : "2017-07-14 15:52+0100" }
> ]
>   }
> ]
>   }
> ]
> treeves@ubuntu:~$ 
> The timestamp in the cluster key, and the regular column, are both truncated 
> to the minute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12683) Batch statement fails with consistency ONE when only 1 node up in DC

[Andy Klages (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143961#comment-16143961
 ] 

Andy Klages commented on CASSANDRA-12683:
-

NetworkTopologyStrategy. The replication factor was set to the number of nodes 
in each DC. So DC1 has an RF of 2 and DC2 has an RF 1. 

> Batch statement fails with consistency ONE when only 1 node up in DC
> 
>
> Key: CASSANDRA-12683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12683
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
> Environment: 3 Cassandra nodes (N1, N2, N3)
> 2 Data Centers (DC1, DC2)
> N1 and N2 members of DC1
> N3 a member of DC2
> 1 keyspace using SimpleReplicationStrategy with RF=3 (can also be 2):
> CREATE KEYSPACE ks WITH 
> REPLICATION={'class':'SimpleStrategy','replication_factor':3};
> 1 column family in the keyspace. For simplicity, a partition key of bigint 
> and one other field of any type. No cluster key needed:
> CREATE TABLE ks.test (
> id  bigint,
> flagboolean,
> PRIMARY KEY(id)
> );
>Reporter: Andy Klages
>Priority: Minor
>
> If Cassandra node N2 is stopped, that only leaves one node (N1) in DC1 
> running. Output from "nodetool status" is:
> Datacenter: DC1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens  OwnsHost ID 
>   Rack
> DN  N2...
> UN  N1...
> Datacenter: DC2
> ==
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens  OwnsHost ID 
>   Rack
> UN  N3...
> The following batch statement will fail when executed on N1 with consistency 
> level of ONE using cqlsh (also fails with Java using Datastax driver):
> CONSISTENCY ONE
> BEGIN BATCH
> UPDATE ks.test SET flag=true where id=1;
> UPDATE ks.test SET flag=true where id=2;
> APPLY BATCH;
> The failure is:
>   File 
> "apache-cassandra-2.1.15/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py",
>  line 3347, in result
> raise self._final_exception
> Unavailable: code=1000 [Unavailable exception] message="Cannot achieve 
> consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 
> 'consistency': 'ONE'}
> There are 2 replicas alive but for some reason, Cassandra thinks there are 
> none.
> If each statement is executed individually (i.e. no batch), each one succeeds.
> This same batch query succeeds on N3, which is the other node in the cluster 
> that is running. My analysis shows this happens when the query is executed on 
> a node in a DC where all other nodes in that DC is down. The batch statement 
> has 2 or more queries with different partition keys. If all of the partition 
> keys are the same value, it succeeds. As the replication factor is set to the 
> number of nodes in the cluster (full replication) and the consistency level 
> is ONE, the batch statement should succeed.
> 2 workarounds for this are:
> 1. Set the consistency level to ANY.
> 2. Use an unlogged batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13807) CircleCI fix - only collect the xml file from containers where it exists

[Eduard Tudenhoefner (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143886#comment-16143886
 ] 

Eduard Tudenhoefner commented on CASSANDRA-13807:
-

[~krummas] changes lgtm, so +1 on merging

> CircleCI fix - only collect the xml file from containers where it exists
> 
>
> Key: CASSANDRA-13807
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13807
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> Followup from CASSANDRA-13775 - my fix with {{ant eclipse-warnings}} 
> obviously does not work since it doesn't generate any xml files
> Push a new fix here: 
> https://github.com/krummas/cassandra/commits/marcuse/fix_circle_3.0 which 
> only collects the xml file from the first 3 containers
> Test running here: https://circleci.com/gh/krummas/cassandra/86



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12870) Calculate pending ranges for identical KS settings just once

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143823#comment-16143823
 ] 

Joel Knighton commented on CASSANDRA-12870:
---

Yes, sorry - this dropped off my radar. I can get to this in the next week or 
so. Otherwise, if someone else is interested in taking it sooner and does so, I 
definitely won't be offended.

> Calculate pending ranges for identical KS settings just once
> 
>
> Key: CASSANDRA-12870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12870
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 4.x
>
> Attachments: 12870-trunk.patch
>
>
> The {{TokenMetadata.calculatePendingRanges()}} operation can be very 
> expensive and already has been subject to micro optimization in 
> CASSANDRA-9258. Instead of further optimizing the calculation itself, I'd 
> like to prevent it from being executed as often by only calling it just once 
> for all keyspaces sharing identical replication settings. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13299) Potential OOMs and lock contention in write path streams

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143793#comment-16143793
 ] 

ZhaoYang commented on CASSANDRA-13299:
--

[~brstgt] could you give some feedback?

> Potential OOMs and lock contention in write path streams
> 
>
> Key: CASSANDRA-13299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13299
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Assignee: ZhaoYang
>
> I see a potential OOM, when a stream (e.g. repair) goes through the write 
> path as it is with MVs.
> StreamReceiveTask gets a bunch of SSTableReaders. These produce rowiterators 
> and they again produce mutations. So every partition creates a single 
> mutation, which in case of (very) big partitions can result in (very) big 
> mutations. Those are created on heap and stay there until they finished 
> processing.
> I don't think it is necessary to create a single mutation for each partition. 
> Why don't we implement a PartitionUpdateGeneratorIterator that takes a 
> UnfilteredRowIterator and a max size and spits out PartitionUpdates to be 
> used to create and apply mutations?
> The max size should be something like min(reasonable_absolute_max_size, 
> max_mutation_size, commitlog_segment_size / 2). reasonable_absolute_max_size 
> could be like 16M or sth.
> A mutation shouldn't be too large as it also affects MV partition locking. 
> The longer a MV partition is locked during a stream, the higher chances are 
> that WTE's occur during streams.
> I could also imagine that a max number of updates per mutation regardless of 
> size in bytes could make sense to avoid lock contention.
> Love to get feedback and suggestions, incl. naming suggestions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13620) Don't skip corrupt sstables on startup

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13620:

   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   4.0
   3.11.1
   3.0.15
   Status: Resolved  (was: Ready to Commit)

Committed as {{dfbe3fabd266493e69}}, thanks

> Don't skip corrupt sstables on startup
> --
>
> Key: CASSANDRA-13620
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13620
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.15, 3.11.1, 4.0
>
> Attachments: 13620-3.0.png, 13620-3.11.png, 13620-trunk.png
>
>
> If we get an IOException when opening an sstable on startup, we just 
> [skip|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L563-L567]
>  it and continue starting
> we should use the DiskFailurePolicy and never explicitly catch an IOException 
> here



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Dont throw IOExceptions in when opening sstables

[marcuse]

Dont throw IOExceptions in when opening sstables

Patch by marcuse; reviewed by Ariel Weisberg for CASSANDRA-13620


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dfbe3fab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dfbe3fab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dfbe3fab

Branch: refs/heads/cassandra-3.11
Commit: dfbe3fabd266493e698c194ef90b4dfc7d62b030
Parents: 95f1b23
Author: Marcus Eriksson 
Authored: Mon Jun 19 15:22:57 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:37:26 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 22 +---
 .../io/sstable/format/SSTableReader.java| 19 +++--
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 4 files changed, 38 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba35152..5ccd5cd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't skip corrupted sstables on startup (CASSANDRA-13620)
  * Fix the merging of cells with different user type versions (CASSANDRA-13776)
  * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
  * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index f720330..7251244 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -59,8 +59,10 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.index.SecondaryIndexManager;
 import org.apache.cassandra.index.internal.CassandraIndex;
 import org.apache.cassandra.index.transactions.UpdateTransaction;
+import org.apache.cassandra.io.FSError;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
+import org.apache.cassandra.io.sstable.CorruptSSTableException;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
 import org.apache.cassandra.io.sstable.format.*;
@@ -691,7 +693,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 }
 catch (IOException e)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(e, 
entry.getKey().filenameFor(Component.STATS)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, e);
 continue;
 }
 
@@ -718,9 +721,22 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 reader = SSTableReader.open(newDescriptor, entry.getValue(), 
metadata);
 }
-catch (IOException e)
+catch (CorruptSSTableException ex)
+{
+FileUtils.handleCorruptSSTable(ex);
+logger.error("Corrupt sstable {}; skipping table", entry, ex);
+continue;
+}
+catch (FSError ex)
+{
+FileUtils.handleFSError(ex);
+logger.error("Cannot read sstable {}; file system error, 
skipping table", entry, ex);
+continue;
+}
+catch (IOException ex)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(ex, 
entry.getKey().filenameFor(Component.DATA)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, ex);
 continue;
 }
 newSSTables.add(reader);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index cd41b5b..d56b3e7 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -446,7 

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/809f3b30
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/809f3b30
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/809f3b30

Branch: refs/heads/cassandra-3.11
Commit: 809f3b30e88b57f2c0dcf71db400710233120143
Parents: 26fedcd dfbe3fa
Author: Marcus Eriksson 
Authored: Mon Aug 28 15:39:49 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:39:49 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 22 +---
 .../io/sstable/format/SSTableReader.java| 19 +++--
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 4 files changed, 38 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/CHANGES.txt
--
diff --cc CHANGES.txt
index 8fb8e2f,5ccd5cd..b0dbd60
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
   * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
   * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/test/unit/org/apache/cassandra/db/ScrubTest.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Dont throw IOExceptions in when opening sstables

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 95f1b23bf -> dfbe3fabd
  refs/heads/cassandra-3.11 26fedcd0b -> 809f3b30e
  refs/heads/trunk 68c2d2e6a -> 326f3a7c7


Dont throw IOExceptions in when opening sstables

Patch by marcuse; reviewed by Ariel Weisberg for CASSANDRA-13620


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dfbe3fab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dfbe3fab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dfbe3fab

Branch: refs/heads/cassandra-3.0
Commit: dfbe3fabd266493e698c194ef90b4dfc7d62b030
Parents: 95f1b23
Author: Marcus Eriksson 
Authored: Mon Jun 19 15:22:57 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:37:26 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 22 +---
 .../io/sstable/format/SSTableReader.java| 19 +++--
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 4 files changed, 38 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba35152..5ccd5cd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't skip corrupted sstables on startup (CASSANDRA-13620)
  * Fix the merging of cells with different user type versions (CASSANDRA-13776)
  * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
  * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index f720330..7251244 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -59,8 +59,10 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.index.SecondaryIndexManager;
 import org.apache.cassandra.index.internal.CassandraIndex;
 import org.apache.cassandra.index.transactions.UpdateTransaction;
+import org.apache.cassandra.io.FSError;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
+import org.apache.cassandra.io.sstable.CorruptSSTableException;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
 import org.apache.cassandra.io.sstable.format.*;
@@ -691,7 +693,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 }
 catch (IOException e)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(e, 
entry.getKey().filenameFor(Component.STATS)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, e);
 continue;
 }
 
@@ -718,9 +721,22 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 reader = SSTableReader.open(newDescriptor, entry.getValue(), 
metadata);
 }
-catch (IOException e)
+catch (CorruptSSTableException ex)
+{
+FileUtils.handleCorruptSSTable(ex);
+logger.error("Corrupt sstable {}; skipping table", entry, ex);
+continue;
+}
+catch (FSError ex)
+{
+FileUtils.handleFSError(ex);
+logger.error("Cannot read sstable {}; file system error, 
skipping table", entry, ex);
+continue;
+}
+catch (IOException ex)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(ex, 
entry.getKey().filenameFor(Component.DATA)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, ex);
 continue;
 }
 newSSTables.add(reader);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[marcuse]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/326f3a7c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/326f3a7c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/326f3a7c

Branch: refs/heads/trunk
Commit: 326f3a7c7d8f20d0389d9cc036b0bb32d37462be
Parents: 68c2d2e 809f3b3
Author: Marcus Eriksson 
Authored: Mon Aug 28 15:40:11 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:41:04 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 19 --
 .../cassandra/io/sstable/SSTableLoader.java |  4 +-
 .../io/sstable/format/SSTableReader.java| 66 +---
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 5 files changed, 62 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/326f3a7c/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/326f3a7c/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 73e7db6,23d0b8e..5aecc9d
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -62,8 -60,10 +62,11 @@@ import org.apache.cassandra.exceptions.
  import org.apache.cassandra.index.SecondaryIndexManager;
  import org.apache.cassandra.index.internal.CassandraIndex;
  import org.apache.cassandra.index.transactions.UpdateTransaction;
+ import org.apache.cassandra.io.FSError;
++import org.apache.cassandra.io.FSReadError;
  import org.apache.cassandra.io.FSWriteError;
  import org.apache.cassandra.io.sstable.Component;
+ import org.apache.cassandra.io.sstable.CorruptSSTableException;
  import org.apache.cassandra.io.sstable.Descriptor;
  import org.apache.cassandra.io.sstable.SSTableMultiWriter;
  import org.apache.cassandra.io.sstable.format.*;
@@@ -781,11 -767,24 +785,18 @@@ public class ColumnFamilyStore implemen
  {
  reader = SSTableReader.open(newDescriptor, entry.getValue(), 
metadata);
  }
- catch (IOException e)
+ catch (CorruptSSTableException ex)
+ {
+ FileUtils.handleCorruptSSTable(ex);
+ logger.error("Corrupt sstable {}; skipping table", entry, ex);
+ continue;
+ }
+ catch (FSError ex)
  {
- SSTableReader.logOpenException(entry.getKey(), e);
+ FileUtils.handleFSError(ex);
+ logger.error("Cannot read sstable {}; file system error, 
skipping table", entry, ex);
  continue;
  }
 -catch (IOException ex)
 -{
 -FileUtils.handleCorruptSSTable(new 
CorruptSSTableException(ex, entry.getKey().filenameFor(Component.DATA)));
 -logger.error("Cannot read sstable {}; other IO error, 
skipping table", entry, ex);
 -continue;
 -}
  newSSTables.add(reader);
  }
  
@@@ -1912,7 -1906,7 +1923,7 @@@
  }
  }
  }
--catch (IOException | RuntimeException e)
++catch (FSReadError | RuntimeException e)
  {
  // In case one of the snapshot sstables fails to open,
  // we must release the references to the ones we opened so far

http://git-wip-us.apache.org/repos/asf/cassandra/blob/326f3a7c/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
index dc56520,043f6fa..69e5c85
--- a/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
@@@ -25,7 -25,7 +25,8 @@@ import java.util.*
  import com.google.common.collect.HashMultimap;
  import com.google.common.collect.Multimap;
  
 -import org.apache.cassandra.config.CFMetaData;
++import org.apache.cassandra.io.FSError;
 +import org.apache.cassandra.schema.TableMetadataRef;
  import org.apache.cassandra.db.Directories;
  import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
  import org.apache.cassandra.dht.Range;
@@@ -138,8 -138,8 +139,9 @@@ public class SSTableLoader implements S
// to conserve heap space when 
bulk loading
sstable.releaseSummary();
}
-- 

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/809f3b30
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/809f3b30
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/809f3b30

Branch: refs/heads/trunk
Commit: 809f3b30e88b57f2c0dcf71db400710233120143
Parents: 26fedcd dfbe3fa
Author: Marcus Eriksson 
Authored: Mon Aug 28 15:39:49 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:39:49 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 22 +---
 .../io/sstable/format/SSTableReader.java| 19 +++--
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 4 files changed, 38 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/CHANGES.txt
--
diff --cc CHANGES.txt
index 8fb8e2f,5ccd5cd..b0dbd60
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
   * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
   * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/809f3b30/test/unit/org/apache/cassandra/db/ScrubTest.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Dont throw IOExceptions in when opening sstables

[marcuse]

Dont throw IOExceptions in when opening sstables

Patch by marcuse; reviewed by Ariel Weisberg for CASSANDRA-13620


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dfbe3fab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dfbe3fab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dfbe3fab

Branch: refs/heads/trunk
Commit: dfbe3fabd266493e698c194ef90b4dfc7d62b030
Parents: 95f1b23
Author: Marcus Eriksson 
Authored: Mon Jun 19 15:22:57 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Aug 28 15:37:26 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 22 +---
 .../io/sstable/format/SSTableReader.java| 19 +++--
 .../unit/org/apache/cassandra/db/ScrubTest.java |  2 +-
 4 files changed, 38 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba35152..5ccd5cd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Don't skip corrupted sstables on startup (CASSANDRA-13620)
  * Fix the merging of cells with different user type versions (CASSANDRA-13776)
  * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
  * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index f720330..7251244 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -59,8 +59,10 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.index.SecondaryIndexManager;
 import org.apache.cassandra.index.internal.CassandraIndex;
 import org.apache.cassandra.index.transactions.UpdateTransaction;
+import org.apache.cassandra.io.FSError;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
+import org.apache.cassandra.io.sstable.CorruptSSTableException;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
 import org.apache.cassandra.io.sstable.format.*;
@@ -691,7 +693,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 }
 catch (IOException e)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(e, 
entry.getKey().filenameFor(Component.STATS)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, e);
 continue;
 }
 
@@ -718,9 +721,22 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 reader = SSTableReader.open(newDescriptor, entry.getValue(), 
metadata);
 }
-catch (IOException e)
+catch (CorruptSSTableException ex)
+{
+FileUtils.handleCorruptSSTable(ex);
+logger.error("Corrupt sstable {}; skipping table", entry, ex);
+continue;
+}
+catch (FSError ex)
+{
+FileUtils.handleFSError(ex);
+logger.error("Cannot read sstable {}; file system error, 
skipping table", entry, ex);
+continue;
+}
+catch (IOException ex)
 {
-SSTableReader.logOpenException(entry.getKey(), e);
+FileUtils.handleCorruptSSTable(new CorruptSSTableException(ex, 
entry.getKey().filenameFor(Component.DATA)));
+logger.error("Cannot read sstable {}; other IO error, skipping 
table", entry, ex);
 continue;
 }
 newSSTables.add(reader);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dfbe3fab/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index cd41b5b..d56b3e7 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -446,7 +446,16 

[jira] [Commented] (CASSANDRA-12696) Allow to change logging levels based on components

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143772#comment-16143772
 ] 

Chris Lohfink commented on CASSANDRA-12696:
---

LGTM, and its a great idea +1

I am a little worried about keeping this in sync as changes and refactors occur 
but I cant think of anything to help with that short of annotating classes or 
something which could be revisited as a followup in the future

> Allow to change logging levels based on components
> --
>
> Key: CASSANDRA-12696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12696
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: lhf
> Attachments: 12696-trunk.patch
>
>
> Currently users are able to dynamically change logging configuration by using 
> {{nodetool setlogginglevel  }}. Unfortunately this requires to 
> know a bit about the Cassandra package hierarchy and gathering all the 
> involved packages/classes can be tedious, especially in troubleshooting 
> situations. What I'd like to have is a way to tell a user to "_when X 
> happens, enable debug logs for bootstrapping/repair/compactions/.._" by 
> simply running e.g. {{nodetool setlogginglevel bootstrap DEBUG}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13786) Validation compactions can cause orphan sstable warnings

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143758#comment-16143758
 ] 

Marcus Eriksson commented on CASSANDRA-13786:
-

makes sense, added 2 comments on the gh branch

> Validation compactions can cause orphan sstable warnings
> 
>
> Key: CASSANDRA-13786
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13786
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> I've seen LevelledCompactionStrategy occasionally logging: 
> {quote} from level 0 is not on corresponding level in the 
> leveled manifest. This is not a problem per se, but may indicate an orphaned 
> sstable due to a failed compaction not cleaned up properly."{quote} warnings 
> from a ValidationExecutor thread.
> What's happening here is that a compaction running concurrently with the 
> validation is promoting (or demoting) sstables as part of an incremental 
> repair, and an sstable has changed hands by the time the validation 
> compaction gets around to getting scanners for it. The sstable 
> isolation/synchronization done by validation compactions is a lot looser than 
> normal compactions, so seeing this happen isn't very surprising. Given that 
> it's harmless, and not unexpected, I think it would be best to not log these 
> during validation compactions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143749#comment-16143749
 ] 

Marcus Eriksson commented on CASSANDRA-13418:
-

bq. N.B: I tried to apply the syle guide found in .idea/codeStyleSettings.xml 
but it is changing me a lot of things. Do you know if it is up to date ?
It is up to date, not sure what you mean by "apply" here?

bq. Could we, instead of setting uncheckedTombstoneCompaction, log a warning 
telling the user that they probably want to uncheckedTombstoneCompaction set as 
well?
+1

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13756) StreamingHistogram is not thread safe

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143747#comment-16143747
 ] 

Hannu Kröger commented on CASSANDRA-13756:
--

Cassandra cleanup is probably also affected by this.

> StreamingHistogram is not thread safe
> -
>
> Key: CASSANDRA-13756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13756
> Project: Cassandra
>  Issue Type: Bug
>Reporter: xiangzhou xia
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x
>
>
> When we test C*3 in shadow cluster, we notice after a period of time, several 
> data node suddenly run into 100% cpu and stop process query anymore.
> After investigation, we found that threads are stuck on the sum() in 
> streaminghistogram class. Those are jmx threads that working on expose 
> getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is 
> a chance that multiple jmx thread is access streaminghistogram at the same 
> time).  
> After further investigation, we find that the optimization in CASSANDRA-13038 
> led to a spool flush every time when we call sum(). Since TreeMap is not 
> thread safe, threads will be stuck when multiple threads visit sum() at the 
> same time.
> There are two approaches to solve this issue. 
> The first one is to add a lock to the flush in sum() which will introduce 
> some extra overhead to streaminghistogram.
> The second one is to avoid streaminghistogram to be access by multiple 
> threads. For our specific case, is to remove the metrics we added.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143730#comment-16143730
 ] 

Jason Brown commented on CASSANDRA-13789:
-

crap ... I'll look into this today.

> Reduce memory copies and object creations when acting on  ByteBufs
> --
>
> Key: CASSANDRA-13789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13789
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Norman Maurer
>Assignee: Norman Maurer
> Fix For: 4.0
>
> Attachments: 
> 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch, 
> 0001-Reduce-memory-copies-and-object-creations-when-actin.patch
>
>
> There are multiple "low-hanging-fruits" when it comes to reduce memory copies 
> and object allocations when acting on ByteBufs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13808) Integer overflows with Amazon Elastic File System (EFS)

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-13808:
--

Assignee: Benjamin Lerer

> Integer overflows with Amazon Elastic File System (EFS)
> ---
>
> Key: CASSANDRA-13808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13808
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Vitalii Ishchenko
>Assignee: Benjamin Lerer
>
> Integer overflow issue was fixed for cassandra 2.2, but since then new 
> property was introduced in config that also derives from disk size  
> {{cdc_total_space_in_mb}}
> It should be updated too 
> https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Reopened] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp reopened CASSANDRA-13789:
--

The change breaks serialization in a couple of places.

Example failure in dtest 
{{cql_tests.py:MiscellaneousCQLTester.prepared_statement_invalidation_test}}.

> Reduce memory copies and object creations when acting on  ByteBufs
> --
>
> Key: CASSANDRA-13789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13789
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Norman Maurer
>Assignee: Norman Maurer
> Fix For: 4.0
>
> Attachments: 
> 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch, 
> 0001-Reduce-memory-copies-and-object-creations-when-actin.patch
>
>
> There are multiple "low-hanging-fruits" when it comes to reduce memory copies 
> and object allocations when acting on ByteBufs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13175) Integrate "Error Prone" Code Analyzer

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13175:

Reviewer: Marcus Eriksson

Just tried it and it surely looks useful

I vote we add it as a new target though, and then run that target on circle-ci. 
Reason I think we should use a new target is that I tried Error Prone 2.1 which 
has a bug making it impossible to build Cassandra 
(https://github.com/google/error-prone/issues/711) - keeping it separate would 
allow us to still build it.

[~spo...@gmail.com] Should we make the error-prone build clean and then commit 
the fixes + the build.xml changes? Or subtasks perhaps?

> Integrate "Error Prone" Code Analyzer
> -
>
> Key: CASSANDRA-13175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13175
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 0001-Add-Error-Prone-code-analyzer.patch, 
> checks-2_2.out, checks-3_0.out, checks-trunk.out
>
>
> I've been playing with [Error Prone|http://errorprone.info/] by integrating 
> it into the build process and to see what kind of warnings it would produce. 
> So far I'm positively impressed by the coverage and usefulness of some of the 
> implemented checks. See attachments for results.
> Unfortunately there are still some issues on how the analyzer is effecting 
> generated code and used guava versions, see 
> [#492|https://github.com/google/error-prone/issues/492]. In case those issues 
> have been solved and the resulting code isn't affected by the analyzer, I'd 
> suggest to add it to trunk with warn only behaviour and some less useful 
> checks disabled. Alternatively a new ant target could be added, maybe with 
> build breaking checks and CI integration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13780) ADD Node streaming throughput performance

[Kevin Rivait (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143687#comment-16143687
 ] 

Kevin Rivait commented on CASSANDRA-13780:
--

yes, we are using vnodes,  num_tokens: 128  on each node
when add a fifth node, we see 4 nodes stream to it
from the system.log
INFO  [main] 2017-08-24 14:16:56,071 StorageService.java:1170 - JOINING: 
Starting to bootstrap...
INFO  [main] 2017-08-24 14:16:56,187 StreamResultFuture.java:87 - [Stream 
#69767f90-88f8-11e7-aa33-f929dc1360c2] Executing streaming plan for Bootstrap
INFO  [StreamConnectionEstablisher:1] 2017-08-24 14:16:56,188 
StreamSession.java:239 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] 
Starting streaming to /10.126.63.127
INFO  [StreamConnectionEstablisher:2] 2017-08-24 14:16:56,188 
StreamSession.java:239 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] 
Starting streaming to /10.126.63.124
INFO  [StreamConnectionEstablisher:3] 2017-08-24 14:16:56,188 
StreamSession.java:239 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] 
Starting streaming to /10.126.63.125
INFO  [StreamConnectionEstablisher:4] 2017-08-24 14:16:56,189 
StreamSession.java:239 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] 
Starting streaming to /10.126.63.121
INFO  [StreamConnectionEstablisher:4] 2017-08-24 14:16:56,196 
StreamCoordinator.java:213 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2, 
ID#0] Beginning stream session with /10.126.63.121
INFO  [StreamConnectionEstablisher:3] 2017-08-24 14:16:56,196 
StreamCoordinator.java:213 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2, 
ID#0] Beginning stream session with /10.126.63.125
INFO  [StreamConnectionEstablisher:2] 2017-08-24 14:16:56,196 
StreamCoordinator.java:213 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2, 
ID#0] Beginning stream session with /10.126.63.124
INFO  [StreamConnectionEstablisher:1] 2017-08-24 14:16:56,196 
StreamCoordinator.java:213 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2, 
ID#0] Beginning stream session with /10.126.63.127
INFO  [STREAM-IN-/10.126.63.121] 2017-08-24 14:16:56,245 
StreamResultFuture.java:169 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2 
ID#0] Prepare completed. Receiving 9 files(1147643092 bytes), sending 0 files(0 
bytes)
INFO  [STREAM-IN-/10.126.63.127] 2017-08-24 14:16:56,245 
StreamResultFuture.java:169 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2 
ID#0] Prepare completed. Receiving 5 files(1354972399 bytes), sending 0 files(0 
bytes)
INFO  [STREAM-IN-/10.126.63.125] 2017-08-24 14:16:56,248 
StreamResultFuture.java:169 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2 
ID#0] Prepare completed. Receiving 9 files(1276409087 bytes), sending 0 files(0 
bytes)
INFO  [STREAM-IN-/10.126.63.124] 2017-08-24 14:16:56,249 
StreamResultFuture.java:169 - [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2 
ID#0] Prepare completed. Receiving 8 files(1446953252 bytes), sending 0 files(0 
bytes)
INFO  [StreamReceiveTask:1] 2017-08-24 14:22:28,495 StreamResultFuture.java:183 
- [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] Session with /10.126.63.121 is 
complete
INFO  [StreamReceiveTask:1] 2017-08-24 14:23:09,001 StreamResultFuture.java:183 
- [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] Session with /10.126.63.125 is 
complete
INFO  [StreamReceiveTask:1] 2017-08-24 14:23:27,289 StreamResultFuture.java:183 
- [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] Session with /10.126.63.127 is 
complete
INFO  [StreamReceiveTask:1] 2017-08-24 14:23:58,065 StreamResultFuture.java:183 
- [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] Session with /10.126.63.124 is 
complete
INFO  [StreamReceiveTask:1] 2017-08-24 14:23:58,068 StreamResultFuture.java:215 
- [Stream #69767f90-88f8-11e7-aa33-f929dc1360c2] All sessions completed

> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 

[jira] [Created] (CASSANDRA-13814) unable to perform DDL and DML operation after install apache-cassandra-2.1.18

[rajasekhar (JIRA)]

rajasekhar created CASSANDRA-13814:
--

 Summary: unable to perform DDL and  DML operation after install 
apache-cassandra-2.1.18
 Key: CASSANDRA-13814
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13814
 Project: Cassandra
  Issue Type: Bug
  Components: Configuration
 Environment: linux(RHEL 6.8)
Reporter: rajasekhar
Priority: Critical
 Fix For: 2.1.18


unable to perform DDL and DML operation after install apache-cassandra-2.1.18. 
please suggest what steps  need to performs and provide the solution step by 
step

*Error:*
InvalidRequest: code=2200 [Invalid query] message="No keyspace has been 
specified. USE a keyspace, or explicitly specify keyspace.tablename"


assandra home path : /u01/Cassandra_home
cassandra bin path: -/u01/Cassandra_home/apache-cassandra-2.1.18/bin
[cassdb@alsc_dev_db bin]$ pwd
/u01/Cassandra_home/apache-cassandra-2.1.18/bin
cassdb@alsc_dev_db bin]$ ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.18 | CQL spec 3.2.1 | Native protocol v3]
Use HELP for help.
cqlsh>
cqlsh> CREATE TABLE test1(sno int);
InvalidRequest: code=2200 [Invalid query] message="No keyspace has been 
specified. USE a keyspace, or explicitly specify keyspace.tablename"
cqlsh>



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[Romain GERARD (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143548#comment-16143548
 ] 

Romain GERARD commented on CASSANDRA-13418:
---

Point noted. 
My only issue with putting a log is that it is easily missable, especially at 
startup.
But as you stated, this is an advance option so we can think that people will 
not stumble-upon it by mistake and will read the doc first before activating 
it. You already have paid the cost for looking for this option, so adding a 
couple instead of one should be painless.
We can go without at first, as you said, as it is the safest default.

@Watchers, feel free to jump on the question

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions

[Dimitar Dimitrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143538#comment-16143538
 ] 

Dimitar Dimitrov edited comment on CASSANDRA-13692 at 8/28/17 9:12 AM:
---

It looks like a good approach here would be to write a test reproducing the 
problem (with one test for each branch throwing a {{RuntimeException}}), then 
apply what looks like the obvious solution (replacing the {{RuntimeExceptions}} 
with {{FSWriteErrors}}) and see if the  failure policy is triggered.
{{org.apache.cassandra.cql3.OutOfSpaceTest}} may be a good starting point, but 
I'll also need to understand better under what conditions exactly is  
{{CompactionAwareWriter#getWriteDirectory(Iterable, long)}} 
triggered.

I'll post another update later today.


was (Author: dimitarndimitrov):
It looks like a good approach here would be to write a test reproducing the 
problem (with one test for each branch throwing a {{RuntimeException}}), then 
apply what looks like the obvious solution (replacing the {{RuntimeExceptions 
}}with {{FSWriteErrors}}) and see if the  failure policy is triggered.
{{org.apache.cassandra.cql3.OutOfSpaceTest}} may be a good starting point, but 
I'll also need to understand better under what conditions exactly is  
{{CompactionAwareWriter#getWriteDirectory(Iterable, long)}} 
triggered.

I'll post another update later today.

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
> return directory;
> }
> {code}
> The fixed code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new FSWriteError(new IOException("Insufficient disk space 
> to write " + writeSize + " bytes"), "");
> return directory;
> }
> {code}
> The fixed code throws FSWE and triggers the failure policy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions

[Dimitar Dimitrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143538#comment-16143538
 ] 

Dimitar Dimitrov commented on CASSANDRA-13692:
--

It looks like a good approach here would be to write a test reproducing the 
problem (with one test for each branch throwing a {{RuntimeException}}), then 
apply what looks like the obvious solution (replacing the {{RuntimeExceptions 
}}with {{FSWriteErrors}}) and see if the  failure policy is triggered.
{{org.apache.cassandra.cql3.OutOfSpaceTest}} may be a good starting point, but 
I'll also need to understand better under what conditions exactly is  
{{CompactionAwareWriter#getWriteDirectory(Iterable, long)}} 
triggered.

I'll post another update later today.

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
> return directory;
> }
> {code}
> The fixed code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new FSWriteError(new IOException("Insufficient disk space 
> to write " + writeSize + " bytes"), "");
> return directory;
> }
> {code}
> The fixed code throws FSWE and triggers the failure policy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13544) Exceptions encountered for concurrent range deletes with mixed cluster keys

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143508#comment-16143508
 ] 

Sylvain Lebresne commented on CASSANDRA-13544:
--

bq. Sylvain Lebresne - any intuition about what this may be a dupe of?

No.

> Exceptions encountered for concurrent range deletes with mixed cluster keys
> ---
>
> Key: CASSANDRA-13544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Cassandra 3.7, Debian Linux
>Reporter: Eric Evans
>
> Using a schema that looks something like...
> {code:sql}
> CREATE TABLE data (
> key text,
> rev int,
> tid timeuuid,
> value blob,
> PRIMARY KEY (key, rev, tid)
> ) WITH CLUSTERING ORDER BY (rev DESC, tid DESC)
> {code}
> ...we are performing range deletes using inequality operators on both {{rev}} 
> and {{tid}} ({{WHERE key = ? AND rev < ?}} and {{WHERE key = ? AND rev = ? 
> AND  tid < ?}}).  These range deletes are interleaved with normal writes 
> probabilistically, and (apparently) when two such range deletes occur 
> concurrently, the following exceptions result.
> {noformat}
> ERROR [SharedPool-Worker-18] 2017-05-19 17:30:22,426 Message.java:611 - 
> Unexpected exception during request; channel = [id: 0x793a853b, 
> L:/10.64.0.36:9042 - R:/10.64.32.112:550
> 48]
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.ClusteringBoundOrBoundary.(ClusteringBoundOrBoundary.java:31)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.ClusteringBoundary.(ClusteringBoundary.java:15) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.ClusteringBoundOrBoundary.inclusiveCloseExclusiveOpen(ClusteringBoundOrBoundary.java:78)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.RangeTombstoneBoundaryMarker.inclusiveCloseExclusiveOpen(RangeTombstoneBoundaryMarker.java:54)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.RangeTombstoneMarker$Merger.merge(RangeTombstoneMarker.java:139)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:521)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:478)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:460)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:320)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:113) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:30) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:53) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:23) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:6) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:735)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:410)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:363)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> 

[jira] [Commented] (CASSANDRA-13792) unable to connect apache-cassandra-3.11.0 after place 4.2.2 JNA jar

[rajasekhar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143503#comment-16143503
 ] 

rajasekhar commented on CASSANDRA-13792:


please update some update to us which we raised the issue.

> unable to connect apache-cassandra-3.11.0 after place 4.2.2 JNA jar 
> 
>
> Key: CASSANDRA-13792
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13792
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: Linux(RHEL 6.8)
>Reporter: rajasekhar
>Priority: Critical
> Fix For: 3.11.0
>
> Attachments: log_cassdb.txt
>
>
> After apply work around below solution, installled the pache-cassandra-3.11 
> but i am not connect server status and cqlsh. please find attached log for 
> more information  and let us know the instatllion completed successfully
> "Jeff Jirsa resolved CASSANDRA-13791.
> 
> Resolution: Duplicate
> This was caused (and resolved) by CASSANDRA-13072 , where JNA was upgraded to 
> {{4.4.0}}. That 4.4.0 JNA library was linked against a different version of 
> glibc ( {{GLIBC_2.14}} ), which causes this error. This is fixed in 
> {{3.11.1}}, but you can work around the issue by downloading the 4.2.2 JNA 
> jar from 
> [here|https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_raw_00a777ec8ab701b843172e23a6cbdc4d6cf48f8d_lib_jna-2D4.2.2.jar=DwICaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=zSydaPm_PKES-H2mB46-9X-iWMDIIymvQCfY02eRW-Q=cSmoTlLQJTt32Gb257uuaz6_WHE6UeXOV5XjO7GTcxk=yieYJ2V_ND3saGBjGH3O-DEwQdfkA44B9yBqtjIU8bE=
>  ] and placing it into the classpath ({{lib/}}), removing {{jna-4.4.0.jar}} 
> in the process.
> > unable to install apache-cassandra-3.11.0 in linux  box
> > ---
> "
> [cassdb@alsc_dev_db bin]$ nodetool status
> -bash: nodetool: command not found
> [cassdb@alsc_dev_db bin]$ pwd
> /u01/Cassandra_home/apache-cassandra-3.11.0/bin
> [cassdb@alsc_dev_db bin]$ sh 
> /u01/Cassandra_home/apache-cassandra-3.11.0/bin/cqlsh
> No appropriate python interpreter found.
> [cassdb@alsc_dev_db bin]$ service cassandra status



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13798) Disallow filtering on non-primary-key base column for MV

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143483#comment-16143483
 ] 

Paulo Motta commented on CASSANDRA-13798:
-

bq. However I'm not convinced about also committing this to 4.0. If we're not 
committing to fixing it in 4.0 then MV's will be pretty useless until 5.0 if it 
does take massive storage changes, which could be a long way away. 

The storage changes required to support this are probably a new SSTable format 
+ changes in the binary protocol, so while it requires a major version, it 
doesn't need to be 5.0, but 4.1 if we are not able to ship this by 4.0.

bq.  Which will be problematic for people who have already gone down this path, 
and we'll be missing a much wanted feature for quite a long time. IMO we made 
the mistake of pushing it out too fast, we should fix it asap.

We should definitely fix this ASAP, but adding this to trunk doesn't prevent 
that in any way, we're just adding this limitation while the problem is not 
fixed. As a follow-up to this, we should create another ticket to implement 
this feature properly. Whether that ticket should block the 4.0 release it's up 
for the community to decide, but on the meantime the code will be correct in 
not allowing people to use broken functionality.

> Disallow filtering on non-primary-key base column for MV
> 
>
> Key: CASSANDRA-13798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13798
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> We should probably consider disallow filtering conditions on non-primary-key 
> base column for Materialized View which is introduced in CASSANDRA-10368.
> The main problem is that the liveness of view row is now depending on 
> multiple base columns (multiple filtered non-pk base column + base column 
> used in view pk) and this semantic could not be properly support without 
> drastic storage format changes. (SEE CASSANDRA-11500, 
> [background|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16119823=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16119823])
> We should step back and re-consider the non-primary-key filtering feature 
> together with supporting multiple non-PK cols in MV clustering key in 
> CASSANDRA-10226.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13812) Missing system keyspace tables are not created

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143479#comment-16143479
 ] 

Sylvain Lebresne commented on CASSANDRA-13812:
--

Ping [~jjirsa] and [~iamaleksey]. It seems using a fixed timestamp of 0 was 
intended by CASSANDRA-13441, and I didn't saw that at the time, but I'm not 
100% of the reasoning. It feels like this basically makes it impossible for use 
to update any parameters on system distributed tables (worst, if we do update 
them, the new values may or may not be picked up depending on how the old and 
new value are resolved (since they will have the same timestamp), which makes 
for an bug that feels easy to go undetected). And in case where whatever new 
value we're set doesn't get picked up, this also mean the code in 
{{StorageService.maybeAddOrUpdateKeyspace}} would try to re-update the table on 
every start without success.

I will note that example in the description is a bit debatable in the sense 
that the fact we actually allow dropping {{system_distributed}} is imo a bug in 
the first place. A bug we should fix and I created CASSANDRA-13813 for that. 
But as said above, even outside that particular case, CASSANDRA-13441 means 
(unless I'm missing something) that we cannot ever do any update to a 
{{system_distributed}} table (we can add stuffs, but we can't update) and that 
doesn't feel ideal to me. Even more so because the restriction is kind of 
silent right now and could be easily overlook in future updates.

> Missing system keyspace tables are not created
> --
>
> Key: CASSANDRA-13812
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13812
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata
>Reporter: ZhaoYang
>
> Auth/Trace/Distributed Keyspaces or Tables dropped are not created on startup 
> although a log message {{MigrationManager.java:220 - Create new table: 
> TableMetadata...}} appears.
> Steps to reproduce:
> # Start node
> # {{DROP TABLE system_distributed.view_build_status;}}
> # {{DROP TABLE system_distributed.repair_history;}}
> # Stop node
> # Start node
> # Tables are *not* created, but log messages appear
> Cause:
> System's keyspaces or tables are created with timestamp 0 in CASSANDRA-13441



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13813) Don't let user drop (or generally break) tables in system_distributed

[Sylvain Lebresne (JIRA)]

Sylvain Lebresne created CASSANDRA-13813:


 Summary: Don't let user drop (or generally break) tables in 
system_distributed
 Key: CASSANDRA-13813
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13813
 Project: Cassandra
  Issue Type: Bug
Reporter: Sylvain Lebresne
 Fix For: 3.0.x, 3.11.x


There is not currently no particular restrictions on schema modifications to 
tables of the {{system_distributed}} keyspace. This does mean you can drop 
those tables, or even alter them in wrong ways like dropping or renaming 
columns. All of which is guaranteed to break stuffs (that is, repair if you 
mess up with on of it's table, or MVs if you mess up with 
{{view_build_status}}).

I'm pretty sure this was never intended and is an oversight of the condition on 
{{ALTERABLE_SYSTEM_KEYSPACES}} in 
[ClientState|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/ClientState.java#L397].
 That condition is such that any keyspace not listed in 
{{ALTERABLE_SYSTEM_KEYSPACES}} (which happens to be the case for 
{{system_distributed}}) has no specific restrictions whatsoever, while given 
the naming it's fair to assume the intention that exactly the opposite.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13743) CAPTURE not easilly usable with PAGING

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143471#comment-16143471
 ] 

Corentin Chary commented on CASSANDRA-13743:


Thanks for merging it and fixing it :)

> CAPTURE not easilly usable with PAGING
> --
>
> Key: CASSANDRA-13743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13743
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 4.0
>
>
> See 
> https://github.com/iksaif/cassandra/commit/7ed56966a7150ced44c375af307685517d7e09a3
>  for a patch fixing that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10496) Make DTCS/TWCS split partitions based on time during compaction

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143470#comment-16143470
 ] 

Corentin Chary commented on CASSANDRA-10496:


I had a patch that would minimize the amount of compactions while trying to 
strictly respect the time windows (and would also make major compaction split 
correctly the sstables). I need to finish it and will try to find time this 
month.

> Make DTCS/TWCS split partitions based on time during compaction
> ---
>
> Key: CASSANDRA-10496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: dtcs
> Fix For: 4.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like 
> [TWCS|CASSANDRA-9666]), we need to split out old data into its own sstable 
> during compaction.
> My initial idea is to just create two sstables, when we create the compaction 
> task we state the start and end times for the window, and any data older than 
> the window will be put in its own sstable.
> By creating a single sstable with old data, we will incrementally get the 
> windows correct - say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now 
> 'correct'. The next compaction would compact in window {{[80, 60]}} and 
> create sstables {{[75]}}, {{[50, 10]}} etc.
> We will probably also want to base the windows on the newest data in the 
> sstables so that we actually have older data than the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13651) Large amount of CPU used by epoll_wait(.., .., .., 0)

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143466#comment-16143466
 ] 

Corentin Chary commented on CASSANDRA-13651:


Great ! I'm good with either bumping netty to this version or merging my patch, 
[~jjirsa] what do you think ?

> Large amount of CPU used by epoll_wait(.., .., .., 0)
> -
>
> Key: CASSANDRA-13651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13651
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 4.x
>
> Attachments: cpu-usage.png
>
>
> I was trying to profile Cassandra under my workload and I kept seeing this 
> backtrace:
> {code}
> epollEventLoopGroup-2-3 State: RUNNABLE CPU usage on sample: 240ms
> io.netty.channel.epoll.Native.epollWait0(int, long, int, int) Native.java 
> (native)
> io.netty.channel.epoll.Native.epollWait(int, EpollEventArray, int) 
> Native.java:111
> io.netty.channel.epoll.EpollEventLoop.epollWait(boolean) 
> EpollEventLoop.java:230
> io.netty.channel.epoll.EpollEventLoop.run() EpollEventLoop.java:254
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run() 
> SingleThreadEventExecutor.java:858
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run() 
> DefaultThreadFactory.java:138
> java.lang.Thread.run() Thread.java:745
> {code}
> At fist I though that the profiler might not be able to profile native code 
> properly, but I wen't further and I realized that most of the CPU was used by 
> {{epoll_wait()}} calls with a timeout of zero.
> Here is the output of perf on this system, which confirms that most of the 
> overhead was with timeout == 0.
> {code}
> Samples: 11M of event 'syscalls:sys_enter_epoll_wait', Event count (approx.): 
> 11594448
> Overhead  Trace output
>   
>  ◆
>   90.06%  epfd: 0x0047, events: 0x7f5588c0c000, maxevents: 0x2000, 
> timeout: 0x   
> ▒
>5.77%  epfd: 0x00b5, events: 0x7fca419ef000, maxevents: 0x1000, 
> timeout: 0x   
> ▒
>1.98%  epfd: 0x00b5, events: 0x7fca419ef000, maxevents: 0x1000, 
> timeout: 0x03e8   
> ▒
>0.04%  epfd: 0x0003, events: 0x2f6af77b9c00, maxevents: 0x0020, 
> timeout: 0x   
> ▒
>0.04%  epfd: 0x002b, events: 0x121ebf63ac00, maxevents: 0x0040, 
> timeout: 0x   
> ▒
>0.03%  epfd: 0x0026, events: 0x7f51f80019c0, maxevents: 0x0020, 
> timeout: 0x   
> ▒
>0.02%  epfd: 0x0003, events: 0x7fe4d80019d0, maxevents: 0x0020, 
> timeout: 0x
> {code}
> Running this time with perf record -ag for call traces:
> {code}
> # Children  Self   sys   usr  Trace output
> 
> #         
> 
> #
>  8.61% 8.61% 0.00% 8.61%  epfd: 0x00a7, events: 
> 0x7fca452d6000, maxevents: 0x1000, timeout: 0x
> |
> ---0x1000200af313
>|  
> --8.61%--0x7fca6117bdac
>   0x7fca60459804
>   epoll_wait
>  2.98% 2.98% 0.00% 2.98%  epfd: 0x00a7, events: 
> 0x7fca452d6000, maxevents: 0x1000, timeout: 0x03e8
> |
> ---0x1000200af313
>0x7fca6117b830
>0x7fca60459804
>epoll_wait
> {code}
> That looks like a lot of CPU used to wait for nothing. I'm not sure if pref 
> reports a per-CPU percentage or a per-system percentage, but that would be 
> still be 10% of the total CPU usage of Cassandra at the minimum.
> I went further and found the code of all that: We schedule a lot of 
> {{Message::Flusher}} with a deadline of 10 usec (5 per messages I think) but 
> netty+epoll only support timeouts above the 

[jira] [Updated] (CASSANDRA-13807) CircleCI fix - only collect the xml file from containers where it exists

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13807:

Reviewer: Eduard Tudenhoefner
  Status: Patch Available  (was: Open)

[~eduard.tudenhoefner] could you have a look?

> CircleCI fix - only collect the xml file from containers where it exists
> 
>
> Key: CASSANDRA-13807
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13807
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> Followup from CASSANDRA-13775 - my fix with {{ant eclipse-warnings}} 
> obviously does not work since it doesn't generate any xml files
> Push a new fix here: 
> https://github.com/krummas/cassandra/commits/marcuse/fix_circle_3.0 which 
> only collects the xml file from the first 3 containers
> Test running here: https://circleci.com/gh/krummas/cassandra/86



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13807) CircleCI fix - only collect the xml file from containers where it exists

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson reassigned CASSANDRA-13807:
---

Assignee: Marcus Eriksson

> CircleCI fix - only collect the xml file from containers where it exists
> 
>
> Key: CASSANDRA-13807
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13807
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> Followup from CASSANDRA-13775 - my fix with {{ant eclipse-warnings}} 
> obviously does not work since it doesn't generate any xml files
> Push a new fix here: 
> https://github.com/krummas/cassandra/commits/marcuse/fix_circle_3.0 which 
> only collects the xml file from the first 3 containers
> Test running here: https://circleci.com/gh/krummas/cassandra/86



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org