[jira] [Created] (CASSANDRA-14229) Separate data drive for smaller SSTable files
Dan Kinder created CASSANDRA-14229: -- Summary: Separate data drive for smaller SSTable files Key: CASSANDRA-14229 URL: https://issues.apache.org/jira/browse/CASSANDRA-14229 Project: Cassandra Issue Type: New Feature Components: Local Write-Read Paths Reporter: Dan Kinder For datasets with an active set of keys that well exceeds ram, it would be quite useful to be able to put certain sstable files (e.g. *-Index.db) on a separate, faster drive(s) than the data. E.g. put the indexes on SSD and data on HDD. Particularly valuable when keys are much smaller than values. Also as ram continues to get more expensive, users that currently optimize by having large key caches may not need to buy as much of it. Our use case is a large dataset like this one. Storing all the data on SSD is cost-prohibitive, and the reads are extremely random (effectively every key is in the active set), so we don't have enough ram to cache it. (I did try using a massive key cache, 64GB, and was seeing strange behavior anyway... irqbalancer process pegged the cpu and the whole thing way underperformed. An investigation for another day.) At the moment our only resolution is to buy enough HDD to handle 2 seeks per read, 1 for the index and 1 for the data. But having indexes on SSD would speed this way up, and practically require us to purchase a small number of SSDs and about 1/2 the number of HDD. One user suggested lvmcache, which could work. I'd like to hear if this will really work optimally and if lvmcache will really keep the right blocks on the faster volume, and how reliable it is at the task. Note: asked about this on the mailing list and it was suggested I create a JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14229) Separate data drive for Index.db files
[ https://issues.apache.org/jira/browse/CASSANDRA-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-14229: --- Summary: Separate data drive for Index.db files (was: Separate data drive for smaller SSTable files) > Separate data drive for Index.db files > -- > > Key: CASSANDRA-14229 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14229 > Project: Cassandra > Issue Type: New Feature > Components: Local Write-Read Paths >Reporter: Dan Kinder >Priority: Minor > > For datasets with an active set of keys that well exceeds ram, it would be > quite useful to be able to put certain sstable files (e.g. *-Index.db) on a > separate, faster drive(s) than the data. E.g. put the indexes on SSD and data > on HDD. Particularly valuable when keys are much smaller than values. Also as > ram continues to get more expensive, users that currently optimize by having > large key caches may not need to buy as much of it. > Our use case is a large dataset like this one. Storing all the data on SSD is > cost-prohibitive, and the reads are extremely random (effectively every key > is in the active set), so we don't have enough ram to cache it. (I did try > using a massive key cache, 64GB, and was seeing strange behavior anyway... > irqbalancer process pegged the cpu and the whole thing way underperformed. An > investigation for another day.) > At the moment our only resolution is to buy enough HDD to handle 2 seeks per > read, 1 for the index and 1 for the data. But having indexes on SSD would > speed this way up, and practically require us to purchase a small number of > SSDs and about 1/2 the number of HDD. > One user suggested lvmcache, which could work. I'd like to hear if this will > really work optimally and if lvmcache will really keep the right blocks on > the faster volume, and how reliable it is at the task. > Note: asked about this on the mailing list and it was suggested I create a > JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288381#comment-16288381
]
Dan Kinder commented on CASSANDRA-13973:
Yep [~jjirsa] that did the trick.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236142#comment-16236142
]
Dan Kinder commented on CASSANDRA-13973:
Thanks [~jjirsa] I'll give it a shot.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216043#comment-16216043
]
Dan Kinder commented on CASSANDRA-13973:
I see... messing with min/max_index_interval wouldn't help would it?
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216010#comment-16216010
]
Dan Kinder commented on CASSANDRA-13973:
The partition key is a string, and there are 4 clustering columns (text, text,
text, timestamp). The partition key and two of the clustering columns are on
average small, on the order of 10 bytes. The third could be larger but have a
maximum of 2048 bytes.
Also, I don't recall running upgradesstables before, if that matters.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215984#comment-16215984
]
Dan Kinder commented on CASSANDRA-13973:
[~jjirsa]
This happens on about 12 out of 16 nodes, much greater than {{RF}}. (No real
need to anonymize the names.)
{noformat}
$ nodetool tablehistograms walker links
walker/links histograms
Percentile SSTables Write Latency Read LatencyPartition Size
Cell Count
(micros) (micros) (bytes)
50% 4.00 51.011155149.91 258
6
75% 5.00 73.463449259.15 1597
42
95% 5.00152.327152383.77 73457
2299
98% 5.00182.798582860.53315852
9887
99% 5.00219.34 53142810.15943127
24601
Min 0.00 8.24 88.1543
0
Max 5.001155149.91 53142810.15 394855527800
962624926
$ nodetool tablestats walker.links
Keyspace: walker
Read Count: 72932
Read Latency: 834.917616670323 ms.
Write Count: 15501219
Write Latency: 0.4392410533003888 ms.
Pending Flushes: 0
Table: links
SSTable count: 14269
SSTables in each level: [0, 9, 68, 349, 2843, 11000, 0, 0, 0]
Space used (live): 5310634842695
Space used (total): 5310634842695
Space used by snapshots (total): 12868175419329
Off heap memory used (total): 16339462
SSTable Compression Ratio: 0.0
Number of partitions (estimate): 20112302
Memtable cell count: 2216835
Memtable data size: 347672840
Memtable off heap memory used: 0
Memtable switch count: 4
Local read count: 80395
Local read latency: 747.603 ms
Local write count: 16891401
Local write latency: 0.118 ms
Pending flushes: 0
Bloom filter false positives: 178
Bloom filter false ratio: 0.00081
Bloom filter space used: 14912616
Bloom filter off heap memory used: 14798464
Index summary off heap memory used: 1540998
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 394855527800
Compacted partition mean bytes: 249068
Average live cells per slice (last five minutes):
11010.451219512195
Maximum live cells per slice (last five minutes): 11864
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
{noformat}
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
>
[jira] [Created] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
Dan Kinder created CASSANDRA-13973:
--
Summary: IllegalArgumentException in upgradesstables compaction
Key: CASSANDRA-13973
URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
Project: Cassandra
Issue Type: Bug
Components: Compaction
Reporter: Dan Kinder
After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try to
run upgradesstables, most of them upgrade fine but I see the exception below on
several nodes, and it doesn't complete.
CASSANDRA-12717 looks similar but the stack trace is not the same, so I assumed
it is not identical. The various nodes this happens on all give the same trace.
Might be notable that this is an analytics cluster with some large partitions,
in the GB size.
{noformat}
error: Out of range: 7316844981
-- StackTrace --
java.lang.IllegalArgumentException: Out of range: 7316844981
at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
at
org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at
org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206598#comment-16206598 ] Dan Kinder commented on CASSANDRA-13923: This seems just as related to CASSANDRA-13948 > Flushers blocked due to many SSTables > - > > Key: CASSANDRA-13923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Local Write-Read Paths > Environment: Cassandra 3.11.0 > Centos 6 (downgraded JNA) > 64GB RAM > 12-disk JBOD >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt > > > This started on the mailing list and I'm not 100% sure of the root cause, > feel free to re-title if needed. > I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving > traffic, thread pools begin to back up and grow pending tasks indefinitely. > This happens to multiple different stages (Read, Mutation) and consistently > builds pending tasks for MemtablePostFlush and MemtableFlushWriter. > Using jstack shows that there is blocking going on when trying to call > getCompactionCandidates, which seems to happen on flush. We have fairly large > nodes that have ~15,000 SSTables per node, all LCS. > I seems like this can cause reads to get blocked because they try to acquire > a read lock when calling shouldDefragment. > And writes, of course, block once we can't allocate anymore memtables, > because flushes are backed up. > We did not have this problem in 2.2.6, so it seems like there is some > regression causing it to be incredibly slow trying to do calls like > getCompactionCandidates that list out the SSTables. > In our case this causes nodes to build up pending tasks and simply stop > responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13948) Avoid deadlock when not able to acquire references for compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202706#comment-16202706
]
Dan Kinder edited comment on CASSANDRA-13948 at 10/13/17 1:04 AM:
--
Just a heads up, I have been seeing these deadlocks happen easily, so I am
running your patch [~pauloricardomg] in addition to
Marcus's[patch|https://github.com/krummas/cassandra/commits/marcuse/13215] from
CASSANDRA-13215.
I do see a large number of "Could not acquire references for compacting
SSTable" happening, in bursts. Will upload a log file.
I also see some of this:
{noformat}
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:344)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:291)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1370)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1326)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:441)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:503)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:121)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:124)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:262)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
{noformat}
UPDATE: the node does successfully complete compactions for a while but
gradually does fewer and fewer and then stops compacting altogether, even
though compactionstats says there are pending compactions:
{noformat}
dkinder@seu-walker-fs01:~$ nodetool compactionstats
pending tasks: 102
- walker.links: 102
dkinder@seu-walker-fs01:~$
{noformat}
It stays this way and periodically prints out some "Could not acquire
references for compacting SSTable" messages. It is able to compact some more if
I restart the node.
tablestats for walker.links:
{noformat}
Keyspace : walker
Read Count: 16952
Read Latency: 10.388668062765454 ms.
Write Count: 277291
Write Latency: 0.0186555207345352 ms.
Pending Flushes: 0
Table: links
SSTable count: 8507
SSTables in each level: [73/4, 81/10, 297/100, 2078/1000, 2402,
3635, 0, 0, 0]
Space used (live): 4902698702556
Space used (total): 4902698702556
Space used by snapshots (total): 13788057680993
Off heap memory used (total): 9835235
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 19996043
Memtable cell count: 360248
Memtable data size: 36729792
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 360248
Local write latency: 0.017 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
[jira] [Updated] (CASSANDRA-13948) Avoid deadlock when not able to acquire references for compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Kinder updated CASSANDRA-13948:
---
Attachment: debug.log
> Avoid deadlock when not able to acquire references for compaction
> -
>
> Key: CASSANDRA-13948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13948
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Fix For: 3.11.x, 4.x
>
> Attachments: debug.log
>
>
> The thread dump below shows a race between an sstable replacement by the
> {{IndexSummaryRedistribution}} and
> {{AbstractCompactionTask.getNextBackgroundTask}}:
> {noformat}
> Thread 94580: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> @bci=1, line=836 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> int) @bci=67, line=870 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> @bci=17, line=1199 (Compiled frame)
> - java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock() @bci=5,
> line=943 (Compiled frame)
> -
> org.apache.cassandra.db.compaction.CompactionStrategyManager.handleListChangedNotification(java.lang.Iterable,
> java.lang.Iterable) @bci=359, line=483 (Interpreted frame)
> -
> org.apache.cassandra.db.compaction.CompactionStrategyManager.handleNotification(org.apache.cassandra.notifications.INotification,
> java.lang.Object) @bci=53, line=555 (Interpreted frame)
> -
> org.apache.cassandra.db.lifecycle.Tracker.notifySSTablesChanged(java.util.Collection,
> java.util.Collection, org.apache.cassandra.db.compaction.OperationType,
> java.lang.Throwable) @bci=50, line=409 (Interpreted frame)
> -
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doCommit(java.lang.Throwable)
> @bci=157, line=227 (Interpreted frame)
> -
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(java.lang.Throwable)
> @bci=61, line=116 (Compiled frame)
> -
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit()
> @bci=2, line=200 (Interpreted frame)
> -
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish()
> @bci=5, line=185 (Interpreted frame)
> -
> org.apache.cassandra.io.sstable.IndexSummaryRedistribution.redistributeSummaries()
> @bci=559, line=130 (Interpreted frame)
> -
> org.apache.cassandra.db.compaction.CompactionManager.runIndexSummaryRedistribution(org.apache.cassandra.io.sstable.IndexSummaryRedistribution)
> @bci=9, line=1420 (Interpreted frame)
> -
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(org.apache.cassandra.io.sstable.IndexSummaryRedistribution)
> @bci=4, line=250 (Interpreted frame)
> -
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries()
> @bci=30, line=228 (Interpreted frame)
> - org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow()
> @bci=4, line=125 (Interpreted frame)
> - org.apache.cassandra.utils.WrappedRunnable.run() @bci=1, line=28
> (Interpreted frame)
> -
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run()
> @bci=4, line=118 (Compiled frame)
> - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
> (Compiled frame)
> - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> @bci=1, line=180 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
> @bci=37, line=294 (Compiled frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1149 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
> (Interpreted frame)
> -
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
> @bci=1, line=81 (Interpreted frame)
> - org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$8.run() @bci=4
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
> {noformat}
> {noformat}
> Thread 94573: (state = IN_JAVA)
> - java.util.HashMap$HashIterator.nextNode() @bci=95, line=1441 (Compiled
> frame;
[jira] [Commented] (CASSANDRA-13948) Avoid deadlock when not able to acquire references for compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202706#comment-16202706
]
Dan Kinder commented on CASSANDRA-13948:
Just a heads up, I have been seeing these deadlocks happen easily, so I am
running your patch [~pauloricardomg] in addition to
Marcus's[patch|https://github.com/krummas/cassandra/commits/marcuse/13215] from
CASSANDRA-13215.
I do see a large number of "Could not acquire references for compacting
SSTables
[BigTableReader(path='/srv/disk2/cassandra-data/walker/links/mc-6566879-big-Data.db')]
which is not a problem per se,unless it happens frequently, in which case it
must be reported. Will retry later." happening, in bursts. Will upload a log
file.
I also see some of this:
{noformat}
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:344)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:291)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1370)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1326)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:441)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:503)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:121)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:124)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:262)
~[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.0.jar:3.11.2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
{noformat}
> Avoid deadlock when not able to acquire references for compaction
> -
>
> Key: CASSANDRA-13948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13948
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Fix For: 3.11.x, 4.x
>
>
> The thread dump below shows a race between an sstable replacement by the
> {{IndexSummaryRedistribution}} and
> {{AbstractCompactionTask.getNextBackgroundTask}}:
> {noformat}
> Thread 94580: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> @bci=1, line=836 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> int) @bci=67, line=870 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> @bci=17, line=1199 (Compiled frame)
> - java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock() @bci=5,
> line=943 (Compiled frame)
> -
> org.apache.cassandra.db.compaction.CompactionStrategyManager.handleListChangedNotification(java.lang.Iterable,
> java.lang.Iterable) @bci=359, line=483
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202502#comment-16202502
]
Dan Kinder commented on CASSANDRA-13943:
Sure, I can try, but I'm not sure how that would help, these nodes have plenty
of cpu and iops available... But there is nonetheless a problem, because
flushers are getting stuck.
Looking at the stack trace, it seems to come down to the readLock in
CompactionStrategyManager. The flushers are blocked on that:
{noformat}
Thread 7385: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int)
@bci=83, line=967 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int)
@bci=10, line=1283 (Compiled frame)
- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5,
line=727 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(org.apache.cassandra.io.sstable.Descriptor,
long, long, org.apache. cassandra.io.sstable.metadata.MetadataCollector,
org.apache.cassandra.db.SerializationHeader, java.util.Collection,
org.apache.cassandra.db.lifecycle. LifecycleTransaction) @bci=4, line=901
(Compiled frame)
-
org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(org.apache.cassandra.io.sstable.Descriptor,
long, long, org.apache.cassandra.io. sstable.metadata.MetadataCollector,
org.apache.cassandra.db.SerializationHeader,
org.apache.cassandra.db.lifecycle.LifecycleTransaction) @bci=21, line=515
(Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(org.apache.cassandra.db.lifecycle.LifecycleTransaction,
java.lang.String, org.apache.cassandra.db.PartitionColumns,
org.apache.cassandra.db.rows.EncodingStats) @bci=104, line=506 (Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.(org.apache.cassandra.db.Memtable,
java.util.concurrent.ConcurrentNavigableMap,
org.apache.cassandra.db.Directories$DataDirectory,
org.apache.cassandra.db.PartitionPosition,
org.apache.cassandra.db.PartitionPosition, org.apache.cassandra.db.lifecycle.
LifecycleTransaction) @bci=253, line=447 (Compiled frame)
-
org.apache.cassandra.db.Memtable$FlushRunnable.(org.apache.cassandra.db.Memtable,
org.apache.cassandra.db.PartitionPosition, org.apache.cassandra.
db.PartitionPosition, org.apache.cassandra.db.Directories$DataDirectory,
org.apache.cassandra.db.lifecycle.LifecycleTransaction) @bci=19, line=417
(Compiled frame)
-
org.apache.cassandra.db.Memtable.createFlushRunnables(org.apache.cassandra.db.lifecycle.LifecycleTransaction)
@bci=125, line=318 (Interpreted frame)
-
org.apache.cassandra.db.Memtable.flushRunnables(org.apache.cassandra.db.lifecycle.LifecycleTransaction)
@bci=2, line=300 (Compiled frame)
-
org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(org.apache.cassandra.db.Memtable,
boolean) @bci=82, line=1137 (Compiled frame)
- org.apache.cassandra.db.ColumnFamilyStore$Flush.run() @bci=85, line=1102
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Compiled frame)
-
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(java.lang.Runnable)
@bci=1, line=81 (Compiled frame)
- org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5.run() @bci=4
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
{noformat}
All the callers in CompactionStrategyManager that *aren't* blocked on a lock
are trying to get background tasks:
{noformat}
Thread 1468: (state = IN_JAVA)
- java.util.HashMap$HashIterator.nextNode() @bci=95, line=1441 (Compiled
frame; information may be imprecise)
- java.util.HashMap$KeyIterator.next() @bci=1, line=1461 (Compiled frame)
- java.util.AbstractCollection.toArray() @bci=39, line=141 (Compiled frame)
- java.util.ArrayList.(java.util.Collection) @bci=6, line=177 (Compiled
frame)
-
org.apache.cassandra.db.compaction.LeveledManifest.ageSortedSSTables(java.util.Collection)
@bci=5, line=731 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(int)
@bci=187, line=644 (Compiled frame)
- org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates()
@bci=290, line=385 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(int)
@bci=4, line=119 (Compiled frame)
-
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Kinder updated CASSANDRA-13943:
---
Attachment: cassandra.yaml
jvm.options
debug.log.zip
debug.log.1.zip
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
No, as far as I can tell no infinite compactions.
I'm doing several hundred mutation per second for one of the tables.
Attached debug logs and configuration. debug.log.1.zip contains node startup.
I also just started seeing some crazy behavior on a node I killed and
restarted... it was adding the same SSTables over and over endlessly:
{noformat}
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,109 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
{noformat}
I took a quick jstack before killing it, attached as
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt,
> cassandra-jstack-2017-10-12.txt, cassandra.yaml, debug.log,
> debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip, jvm.options
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202237#comment-16202237
]
Dan Kinder edited comment on CASSANDRA-13943 at 10/12/17 5:24 PM:
--
[~krummas] after a night I am seeing most nodes start to build up
MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter
Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on
MutationStage and grown lots of pending tasks in that, I assume because
flushing is blocked and there's nowhere else to write.
Attached a stack dump from one of these heavily blocked nodes.
Also notable, since it seems like the blocking is in
{{LeveledManifest.getCompactionCandidates()}}, here is what the SSTable levels
look like for the biggest table:
52 SSTable Level: 0
75 SSTable Level: 1
122 SSTable Level: 2
360 SSTable Level: 3
2724 SSTable Level: 4
10647 SSTable Level: 5
Several other tables have comparable number of SSTables in L0 (20-40 or so),
but those only have a few hundred or a few thousand SSTables.
was (Author: dkinder):
[~krummas] after a night I am seeing most nodes start to build up
MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter
Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on
MutationStage and grown lots of pending tasks in that, I assume because
flushing is blocked and there's nowhere else to write.
Attached a stack dump from one of these heavily blocked nodes.
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12.txt, debug.log,
> debug.log-with-commit-d8f3f2780
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: cassandra-jstack-2017-10-12.txt [~krummas] after a night I am seeing most nodes start to build up MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on MutationStage and grown lots of pending tasks in that, I assume because flushing is blocked and there's nowhere else to write. Attached a stack dump from one of these heavily blocked nodes. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-2017-10-12.txt, debug.log, > debug.log-with-commit-d8f3f2780 > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201286#comment-16201286 ] Dan Kinder commented on CASSANDRA-13943: I've rolled out https://github.com/krummas/cassandra/commits/marcuse/13215 now per the last comment on https://issues.apache.org/jira/browse/CASSANDRA-13215 and am currently running relocatesstables. So far it is not doing the infinite compaction thing. However it's a bit painful, it seems like nodetool compactionstats and nodetool cfstats no longer work, they hang indefinitely for me. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log, debug.log-with-commit-d8f3f2780 > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: debug.log-with-commit-d8f3f2780 Okay, stood up a node on this commit: https://github.com/krummas/cassandra/commit/d8f3f2780fbb9e789f90b095d1f109ebb16f46ff and attached the log. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log, debug.log-with-commit-d8f3f2780 > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199363#comment-16199363 ] Dan Kinder commented on CASSANDRA-13943: Hm, it looks like the new patch though does not change any behavior, and without any changes I don't think my nodes will be able to do any flushing... I might try it on just one node and send those logs. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198777#comment-16198777 ] Dan Kinder commented on CASSANDRA-13943: [~krummas] it looks like this latest past does not include the changes from the simple-cache patch, I'm assuming I should leave that one applied? I.e. use both patches? > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198778#comment-16198778 ] Dan Kinder commented on CASSANDRA-13943: Ope didn't see your message. Got it. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198769#comment-16198769 ] Dan Kinder commented on CASSANDRA-13943: Yeah I am running with the patch from https://issues.apache.org/jira/browse/CASSANDRA-13215 I'll try that latest patch today. Thanks [~krummas] > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197459#comment-16197459
]
Dan Kinder edited comment on CASSANDRA-13943 at 10/9/17 11:45 PM:
--
I do see a questionable {{startsWith}} in a few places:
https://github.com/apache/cassandra/blob/ba87ab4e954ad2e537f6690953bd7ebaa069f5cd/src/java/org/apache/cassandra/db/Directories.java#L281
https://github.com/apache/cassandra/blob/7d4d1a32581ff40ed1049833631832054bcf2316/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java#L309
https://github.com/apache/cassandra/blob/3cec208c40b85e1be0ff8c68fc9d9017945a1ed8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L570
was (Author: dkinder):
I do see a questionable {{startsWith}} here:
https://github.com/apache/cassandra/blob/7d4d1a32581ff40ed1049833631832054bcf2316/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java#L309
Also here:
https://github.com/apache/cassandra/blob/3cec208c40b85e1be0ff8c68fc9d9017945a1ed8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L570
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: debug.log
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197682#comment-16197682
]
Dan Kinder commented on CASSANDRA-13943:
FYI:
{noformat}
data_file_directories:
- /srv/disk1/cassandra-data
- /srv/disk2/cassandra-data
- /srv/disk3/cassandra-data
- /srv/disk4/cassandra-data
- /srv/disk5/cassandra-data
- /srv/disk6/cassandra-data
- /srv/disk7/cassandra-data
- /srv/disk8/cassandra-data
- /srv/disk9/cassandra-data
- /srv/disk10/cassandra-data
- /srv/disk11/cassandra-data
- /srv/disk12/cassandra-data
{noformat}
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: debug.log
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197459#comment-16197459
]
Dan Kinder commented on CASSANDRA-13943:
I do see a questionable {{startsWith}} here:
https://github.com/apache/cassandra/blob/7d4d1a32581ff40ed1049833631832054bcf2316/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java#L309
Also here:
https://github.com/apache/cassandra/blob/3cec208c40b85e1be0ff8c68fc9d9017945a1ed8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L570
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: debug.log
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: debug.log > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
Dan Kinder created CASSANDRA-13943: -- Summary: Infinite compaction of L0 SSTables in JBOD Key: CASSANDRA-13943 URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 Project: Cassandra Issue Type: Bug Components: Compaction Environment: Cassandra 3.11.0 / Centos 6 Reporter: Dan Kinder I recently upgraded from 2.2.6 to 3.11.0. I am seeing Cassandra loop infinitely compacting the same data over and over. Attaching logs. It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It does create new SSTables but immediately recompacts again. Note that I am not inserting anything at the moment, there is no flushing happening on this table (Memtable switch count has not changed). My theory is that it somehow thinks those should be compaction candidates. But they shouldn't be, they are on different disks and I ran nodetool relocatesstables as well as nodetool compact. So, it tries to compact them together, but the compaction results in the exact same 2 SSTables on the 2 disks, because the keys are split by data disk. This is pretty serious, because all our nodes right now are consuming CPU doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191699#comment-16191699 ] Dan Kinder commented on CASSANDRA-13923: So far the nodes seem to be performing better, they are doing a lot of compactions. So, that patch seemed to do the trick. Sadly I was not able to get a jstack with -l. > Flushers blocked due to many SSTables > - > > Key: CASSANDRA-13923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Local Write-Read Paths > Environment: Cassandra 3.11.0 > Centos 6 (downgraded JNA) > 64GB RAM > 12-disk JBOD >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt > > > This started on the mailing list and I'm not 100% sure of the root cause, > feel free to re-title if needed. > I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving > traffic, thread pools begin to back up and grow pending tasks indefinitely. > This happens to multiple different stages (Read, Mutation) and consistently > builds pending tasks for MemtablePostFlush and MemtableFlushWriter. > Using jstack shows that there is blocking going on when trying to call > getCompactionCandidates, which seems to happen on flush. We have fairly large > nodes that have ~15,000 SSTables per node, all LCS. > I seems like this can cause reads to get blocked because they try to acquire > a read lock when calling shouldDefragment. > And writes, of course, block once we can't allocate anymore memtables, > because flushes are backed up. > We did not have this problem in 2.2.6, so it seems like there is some > regression causing it to be incredibly slow trying to do calls like > getCompactionCandidates that list out the SSTables. > In our case this causes nodes to build up pending tasks and simply stop > responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189997#comment-16189997 ] Dan Kinder commented on CASSANDRA-13923: Brought up one node with [the patch|https://issues.apache.org/jira/secure/attachment/12852352/simple-cache.patch] -- wow, it starts up WAY faster. It also seemed to properly complete flushes on that node so I'm pushing out to other nodes now to see how it does. > Flushers blocked due to many SSTables > - > > Key: CASSANDRA-13923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Local Write-Read Paths > Environment: Cassandra 3.11.0 > Centos 6 (downgraded JNA) > 64GB RAM > 12-disk JBOD >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt > > > This started on the mailing list and I'm not 100% sure of the root cause, > feel free to re-title if needed. > I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving > traffic, thread pools begin to back up and grow pending tasks indefinitely. > This happens to multiple different stages (Read, Mutation) and consistently > builds pending tasks for MemtablePostFlush and MemtableFlushWriter. > Using jstack shows that there is blocking going on when trying to call > getCompactionCandidates, which seems to happen on flush. We have fairly large > nodes that have ~15,000 SSTables per node, all LCS. > I seems like this can cause reads to get blocked because they try to acquire > a read lock when calling shouldDefragment. > And writes, of course, block once we can't allocate anymore memtables, > because flushes are backed up. > We did not have this problem in 2.2.6, so it seems like there is some > regression causing it to be incredibly slow trying to do calls like > getCompactionCandidates that list out the SSTables. > In our case this causes nodes to build up pending tasks and simply stop > responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189886#comment-16189886 ] Dan Kinder commented on CASSANDRA-13923: I did notice that startup time was significantly slower now on 3.11, similar to the symptom reported in https://issues.apache.org/jira/browse/CASSANDRA-13215 I'll try to get you the other info shortly and apply that patch. Unfortunately jstack sometimes NPEs, so once I get lucky you'll get one with -l > Flushers blocked due to many SSTables > - > > Key: CASSANDRA-13923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Local Write-Read Paths > Environment: Cassandra 3.11.0 > Centos 6 (downgraded JNA) > 64GB RAM > 12-disk JBOD >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt > > > This started on the mailing list and I'm not 100% sure of the root cause, > feel free to re-title if needed. > I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving > traffic, thread pools begin to back up and grow pending tasks indefinitely. > This happens to multiple different stages (Read, Mutation) and consistently > builds pending tasks for MemtablePostFlush and MemtableFlushWriter. > Using jstack shows that there is blocking going on when trying to call > getCompactionCandidates, which seems to happen on flush. We have fairly large > nodes that have ~15,000 SSTables per node, all LCS. > I seems like this can cause reads to get blocked because they try to acquire > a read lock when calling shouldDefragment. > And writes, of course, block once we can't allocate anymore memtables, > because flushes are backed up. > We did not have this problem in 2.2.6, so it seems like there is some > regression causing it to be incredibly slow trying to do calls like > getCompactionCandidates that list out the SSTables. > In our case this causes nodes to build up pending tasks and simply stop > responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188831#comment-16188831 ] Dan Kinder commented on CASSANDRA-13923: >From my own poking around, >https://issues.apache.org/jira/browse/CASSANDRA-9882 seemed related. > Flushers blocked due to many SSTables > - > > Key: CASSANDRA-13923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Local Write-Read Paths > Environment: Cassandra 3.11.0 > Centos 6 (downgraded JNA) > 64GB RAM > 12-disk JBOD >Reporter: Dan Kinder > Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt > > > This started on the mailing list and I'm not 100% sure of the root cause, > feel free to re-title if needed. > I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving > traffic, thread pools begin to back up and grow pending tasks indefinitely. > This happens to multiple different stages (Read, Mutation) and consistently > builds pending tasks for MemtablePostFlush and MemtableFlushWriter. > Using jstack shows that there is blocking going on when trying to call > getCompactionCandidates, which seems to happen on flush. We have fairly large > nodes that have ~15,000 SSTables per node, all LCS. > I seems like this can cause reads to get blocked because they try to acquire > a read lock when calling shouldDefragment. > And writes, of course, block once we can't allocate anymore memtables, > because flushes are backed up. > We did not have this problem in 2.2.6, so it seems like there is some > regression causing it to be incredibly slow trying to do calls like > getCompactionCandidates that list out the SSTables. > In our case this causes nodes to build up pending tasks and simply stop > responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13923) Flushers blocked due to many SSTables
Dan Kinder created CASSANDRA-13923: -- Summary: Flushers blocked due to many SSTables Key: CASSANDRA-13923 URL: https://issues.apache.org/jira/browse/CASSANDRA-13923 Project: Cassandra Issue Type: Bug Components: Compaction, Local Write-Read Paths Environment: Cassandra 3.11.0 Centos 6 (downgraded JNA) 64GB RAM 12-disk JBOD Reporter: Dan Kinder Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt This started on the mailing list and I'm not 100% sure of the root cause, feel free to re-title if needed. I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic, thread pools begin to back up and grow pending tasks indefinitely. This happens to multiple different stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush and MemtableFlushWriter. Using jstack shows that there is blocking going on when trying to call getCompactionCandidates, which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per node, all LCS. I seems like this can cause reads to get blocked because they try to acquire a read lock when calling shouldDefragment. And writes, of course, block once we can't allocate anymore memtables, because flushes are backed up. We did not have this problem in 2.2.6, so it seems like there is some regression causing it to be incredibly slow trying to do calls like getCompactionCandidates that list out the SSTables. In our case this causes nodes to build up pending tasks and simply stop responding to requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11904) Exception in thread Thread[CompactionExecutor:13358,1,main] java.lang.AssertionError: Memory was freed
[
https://issues.apache.org/jira/browse/CASSANDRA-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359166#comment-15359166
]
Dan Kinder commented on CASSANDRA-11904:
Sorry I don't have any other logs. We had a bunch of issues aside from this one
so we simply rolled it back and didn't investigate debug logs.
> Exception in thread Thread[CompactionExecutor:13358,1,main]
> java.lang.AssertionError: Memory was freed
> --
>
> Key: CASSANDRA-11904
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11904
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Valentin Martinjuk
>Assignee: Marcus Eriksson
> Attachments: system.log.2016-06-10_0514
>
>
> We have Cassandra cluster 2.2.5 with two datacenters(3 nodes each).
> We observe ERRORs below on all nodes. The ERROR is repeated every minute.
> No any complains from customers. Do we have any chance to fix it without
> restart?
> {code}
> ERROR [CompactionExecutor:13996] 2016-05-26 21:20:46,700
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:412)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:88)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:250)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> ERROR [CompactionExecutor:13996] 2016-05-26 21:21:46,702
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
>
[jira] [Updated] (CASSANDRA-11904) Exception in thread Thread[CompactionExecutor:13358,1,main] java.lang.AssertionError: Memory was freed
[
https://issues.apache.org/jira/browse/CASSANDRA-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Kinder updated CASSANDRA-11904:
---
Attachment: system.log.2016-06-10_0514
Attached a log, search for "Memory was freed"
Note that this was on upgrade from 2.2.5 to 3.6. I can't do any further
experimentation now because we had too many issues and gave up on this upgrade
and stuck with 2.2.6. (Unfortunate, because we really wanted the jbod
improvements introduced in 3.2.)
> Exception in thread Thread[CompactionExecutor:13358,1,main]
> java.lang.AssertionError: Memory was freed
> --
>
> Key: CASSANDRA-11904
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11904
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Valentin Martinjuk
>Assignee: Marcus Eriksson
> Attachments: system.log.2016-06-10_0514
>
>
> We have Cassandra cluster 2.2.5 with two datacenters(3 nodes each).
> We observe ERRORs below on all nodes. The ERROR is repeated every minute.
> No any complains from customers. Do we have any chance to fix it without
> restart?
> {code}
> ERROR [CompactionExecutor:13996] 2016-05-26 21:20:46,700
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:412)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:88)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:250)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> ERROR [CompactionExecutor:13996] 2016-05-26 21:21:46,702
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
>
[jira] [Commented] (CASSANDRA-11904) Exception in thread Thread[CompactionExecutor:13358,1,main] java.lang.AssertionError: Memory was freed
[
https://issues.apache.org/jira/browse/CASSANDRA-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322869#comment-15322869
]
Dan Kinder commented on CASSANDRA-11904:
I am also seeing this exception, would be helpful to know if this causes any
issues. This happened on upgrade from 2.2.5 to Cassandra 3.5.0.
> Exception in thread Thread[CompactionExecutor:13358,1,main]
> java.lang.AssertionError: Memory was freed
> --
>
> Key: CASSANDRA-11904
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11904
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Valentin Martinjuk
>
> We have Cassandra cluster 2.2.5 with two datacenters(3 nodes each).
> We observe ERRORs below on all nodes. The ERROR is repeated every minute.
> No any complains from customers. Do we have any chance to fix it without
> restart?
> {code}
> ERROR [CompactionExecutor:13996] 2016-05-26 21:20:46,700
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:412)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:88)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:250)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> ERROR [CompactionExecutor:13996] 2016-05-26 21:21:46,702
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
>
[jira] [Created] (CASSANDRA-11270) sstablerepairedset loads CLASSPATH improperly (RHEL)
Dan Kinder created CASSANDRA-11270:
--
Summary: sstablerepairedset loads CLASSPATH improperly (RHEL)
Key: CASSANDRA-11270
URL: https://issues.apache.org/jira/browse/CASSANDRA-11270
Project: Cassandra
Issue Type: Bug
Components: Tools
Environment: Centos 6.7, Cassandra 2.2.5
Reporter: Dan Kinder
When I try to run sstablerepairedset I get the following:
Error: Could not find or load main class
org.apache.cassandra.tools.SSTableRepairedAtSetter
The tool seems to assume jar files are in {{`dirname $0`/../../lib/*.jar}} but
the tool is in {{/usr/bin/}} so assembling the CLASSPATH appears to be the
failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8816) Dropping Keyspace too slow on Windows
[
https://issues.apache.org/jira/browse/CASSANDRA-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592603#comment-14592603
]
Dan Kinder commented on CASSANDRA-8816:
---
I am seeing the same on one of our Centos 6 boxes. According to the log
Cassandra is taking 10s flushing a whole bunch of memtables, which is a bit
disruptive for our own unit tests.
Dropping Keyspace too slow on Windows
-
Key: CASSANDRA-8816
URL: https://issues.apache.org/jira/browse/CASSANDRA-8816
Project: Cassandra
Issue Type: Improvement
Reporter: Philip Thompson
Assignee: Joshua McKenzie
Priority: Trivial
Labels: Windows
Fix For: 3.x
The dtests
{{secondary_indexes_test.py:TestSecondaryIndexes.test_6924_dropping_ks}} and
{{secondary_indexes_test.py:TestSecondaryIndexes.test_6924_dropping_cf}} are
failing on 2.1-head and trunk on windows. They are running into the client
side 10s timeout when dropping a keyspace.
The tests create and drop the same keyspace 10 times, and only fourth or
fifth iterations of this loop run into the timeout. Raising the timeout to
20s solves the issue. It is not occurring on linux. The keyspace is still
being dropped correctly, so this is labeled 'Improvement'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8961) Data rewrite case causes almost non-functional compaction
Dan Kinder created CASSANDRA-8961:
-
Summary: Data rewrite case causes almost non-functional compaction
Key: CASSANDRA-8961
URL: https://issues.apache.org/jira/browse/CASSANDRA-8961
Project: Cassandra
Issue Type: Bug
Environment: Centos 6.6, Cassandra 2.0.12 (Also seen in Cassandra 2.1)
Reporter: Dan Kinder
Priority: Minor
There seems to be a bug of some kind where compaction grinds to a halt in this
use case: from time to time we have a set of rows we need to migrate,
changing their primary key by deleting the row and inserting a new row with the
same partition key and different cluster key. The python script below
demonstrates this; it takes a bit of time to run (didn't try to optimize it)
but when it's done it will be trying to compact a few hundred megs of data for
a long time... on the order of days, or it will never finish.
Not verified by this sandboxed experiment but it seems that compression
settings do not matter and that this seems to happen to STCS as well, not just
LCS. I am still testing if other patterns cause this terrible compaction
performance, like deleting all rows then inserting or vice versa.
Even if it isn't a bug per se, is there a way to fix or work around this
behavior?
{code}
import string
import random
from cassandra.cluster import Cluster
cluster = Cluster(['localhost'])
db = cluster.connect('walker')
db.execute(DROP KEYSPACE IF EXISTS trial)
db.execute(CREATE KEYSPACE trial
WITH REPLICATION = { 'class': 'SimpleStrategy',
'replication_factor': 1 })
db.execute(CREATE TABLE trial.tbl (
pk text,
data text,
PRIMARY KEY(pk, data)
) WITH compaction = { 'class' : 'LeveledCompactionStrategy' }
AND compression = {'sstable_compression': ''})
# Number of rows to insert and move
n = 20
# Insert n rows with the same partition key, 1KB of unique data in cluster key
for i in range(n):
db.execute(INSERT INTO trial.tbl (pk, data) VALUES ('thepk', %s),
[str(i).zfill(1024)])
# Update those n rows, deleting each and replacing with a very similar row
for i in range(n):
val = str(i).zfill(1024)
db.execute(DELETE FROM trial.tbl WHERE pk = 'thepk' AND data = %s, [val])
db.execute(INSERT INTO trial.tbl (pk, data) VALUES ('thepk', %s), [1 +
val])
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8961) Data rewrite case causes almost non-functional compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359540#comment-14359540
]
Dan Kinder commented on CASSANDRA-8961:
---
I see. Is there some way to make this DELETE query not use RangeTombstones?
Would it work to insert the full set of columns (ex. DELETE pk, data FROM ...)?
Also CASSANDRA-6446 seems related.
Data rewrite case causes almost non-functional compaction
-
Key: CASSANDRA-8961
URL: https://issues.apache.org/jira/browse/CASSANDRA-8961
Project: Cassandra
Issue Type: Bug
Environment: Centos 6.6, Cassandra 2.0.12 (Also seen in Cassandra 2.1)
Reporter: Dan Kinder
Priority: Minor
There seems to be a bug of some kind where compaction grinds to a halt in
this use case: from time to time we have a set of rows we need to migrate,
changing their primary key by deleting the row and inserting a new row with
the same partition key and different cluster key. The python script below
demonstrates this; it takes a bit of time to run (didn't try to optimize it)
but when it's done it will be trying to compact a few hundred megs of data
for a long time... on the order of days, or it will never finish.
Not verified by this sandboxed experiment but it seems that compression
settings do not matter and that this seems to happen to STCS as well, not
just LCS. I am still testing if other patterns cause this terrible compaction
performance, like deleting all rows then inserting or vice versa.
Even if it isn't a bug per se, is there a way to fix or work around this
behavior?
{code}
import string
import random
from cassandra.cluster import Cluster
cluster = Cluster(['localhost'])
db = cluster.connect('walker')
db.execute(DROP KEYSPACE IF EXISTS trial)
db.execute(CREATE KEYSPACE trial
WITH REPLICATION = { 'class': 'SimpleStrategy',
'replication_factor': 1 })
db.execute(CREATE TABLE trial.tbl (
pk text,
data text,
PRIMARY KEY(pk, data)
) WITH compaction = { 'class' : 'LeveledCompactionStrategy' }
AND compression = {'sstable_compression': ''})
# Number of rows to insert and move
n = 20
# Insert n rows with the same partition key, 1KB of unique data in cluster key
for i in range(n):
db.execute(INSERT INTO trial.tbl (pk, data) VALUES ('thepk', %s),
[str(i).zfill(1024)])
# Update those n rows, deleting each and replacing with a very similar row
for i in range(n):
val = str(i).zfill(1024)
db.execute(DELETE FROM trial.tbl WHERE pk = 'thepk' AND data = %s,
[val])
db.execute(INSERT INTO trial.tbl (pk, data) VALUES ('thepk', %s), [1
+ val])
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8892) Read after write inconsistent even on single-node cluster
Dan Kinder created CASSANDRA-8892:
-
Summary: Read after write inconsistent even on single-node cluster
Key: CASSANDRA-8892
URL: https://issues.apache.org/jira/browse/CASSANDRA-8892
Project: Cassandra
Issue Type: Bug
Environment: Centos 6.6, Cassandra 2.0.12
Reporter: Dan Kinder
Priority: Minor
Posted on mailing list, original email:
I had been having the same problem as in those older post:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201411.mbox/%3CCAORswtz+W4Eg2CoYdnEcYYxp9dARWsotaCkyvS5M7+Uo6HT1=a...@mail.gmail.com%3E
To summarize it, on my local box with just one cassandra node I can update and
then select the updated row and get an incorrect response.
My understanding is this may have to do with not having fine-grained enough
timestamp resolution, but regardless I'm wondering: is this actually a bug or
is there any way to mitigate it? It causes sporadic failures in our unit tests,
and having to Sleep() between tests isn't ideal. At least confirming it's a bug
would be nice though.
For those interested, here's a little go program that can reproduce the issue.
When I run it I typically see:
{noformat}
Expected 100 but got: 99
Expected 1000 but got: 999
{noformat}
--- main.go: ---
{code}
package main
import (
fmt
github.com/gocql/gocql
)
func main() {
cf := gocql.NewCluster(localhost)
db, _ := cf.CreateSession()
// Keyspace ut = update test
err := db.Query(`CREATE KEYSPACE IF NOT EXISTS ut
WITH REPLICATION = {'class': 'SimpleStrategy',
'replication_factor': 1 }`).Exec()
if err != nil {
panic(err.Error())
}
err = db.Query(CREATE TABLE IF NOT EXISTS ut.test (key text, val text,
PRIMARY KEY(key))).Exec()
if err != nil {panic(err.Error())
}
err = db.Query(TRUNCATE ut.test).Exec()
if err != nil {
panic(err.Error())
}
err = db.Query(INSERT INTO ut.test (key) VALUES ('foo')).Exec()
if err != nil {
panic(err.Error())
}
for i := 0; i 1; i++ {
val := fmt.Sprintf(%d, i)
db.Query(UPDATE ut.test SET val = ? WHERE key = 'foo', val).Exec()
var result string
db.Query(SELECT val FROM ut.test WHERE key = 'foo').Scan(result)
if result != val {
fmt.Printf(Expected %v but got: %v\n, val, result)
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
