[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146131#comment-16146131
]
Jay Zhuang commented on CASSANDRA-13785:
Thanks [~jjirsa] it's the same issue as CASSANDRA-12014. Closing as duplicated.
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146112#comment-16146112
]
Jeff Jirsa commented on CASSANDRA-13785:
FWIW, there's a related CASSANDRA-12014 which is patch available which TRIES to
guess when this will happen and change the index interval to compensate.
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145915#comment-16145915
]
Jay Zhuang commented on CASSANDRA-13785:
[~aweisberg] Thanks for the quick response.
{quote}
So you are the unlucky person to catch SafeMemoryWriter misbehaving. If we want
our test hygiene to be good you would be the person to write a basic set of
unit tests for it. Go through each external method in SafeMemoryWriter and test
it's API (null handling, largest value, smallest value, negative value, 0) and
internal state (0 length, empty buffer, full buffer, etc.).
{quote}
Make sense, I'll add more unittests.
{quote}
This will still hit the same issue if the decorated key size is larger than
expected right? I know almost everyone uses fixed length keys, but it looks
fragile to not at least fail fast if they use variable length keys or if the
size changes in future iterations.
{quote}
Yes :( It will fail the assert here: {{[assert entries.length() <=
Integer.MAX_VALUE;|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L172]}}
and trigger the same problem.
{quote}
This is a bug in SafeMemoryWriter.length()?
{quote}
I don't think so, maybe it's a bad name.
{{[length()|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L81]}}
means the current position. But if the memory longer than Int.MAX_VALUE, it
moves the length() forward to only cover the Int.MAX_VALUE length, so we can
only use the last Int.MAX_VALUE size. e.g. if memory size is {{10 +
Int.MAX_VALUE}}, then the first length() is 10, if memory size is {{10 + 2 *
Int.MAX_VALUE}}, the first length() is {{10 + Int.MAX_VALUE}}. I think the
purpose is to support Int.MAX_VALUE memory, but not sure why, as it only used
by {{IndexSummaryBuilder}}.
{quote}
Even if this throws AssertionError it still shouldn't leak memory right? We
should log the error, but not leak. I think (I'll check) that we don't treat
assertion failures as fatal and crash the process.
{quote}
It doesn't log the error, the compaction thread just fails silently. Here is
the call stack (it's 3.0.14 with a few debugging code, so the line number might
not exactly match):
{{noformat}}
at org.apache.cassandra.io.util.Memory.(Memory.java:82)
at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.(SafeMemoryWriter.java:31)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.(IndexSummaryBuilder.java:112)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.(BigTableWriter.java:377)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:84)
at
org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93)
at
org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96)
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:64)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:128)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:108)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at java.lang.Thread.run(Thread.java:745)
{{noformat}}
I think the leak should be because the {{writer.finish()}} is not called:
[CompactionTask.java:206|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java#L206]
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL:
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145799#comment-16145799
]
Ariel Weisberg commented on CASSANDRA-13785:
New stuff and summarizing.
* So you are the unlucky person to catch SafeMemoryWriter misbehaving. If we
want our test hygiene to be good you would be the person to write a basic set
of unit tests for it. Go through each external method in SafeMemoryWriter and
test it's API (null handling, largest value, smallest value, negative value, 0)
and internal state (0 length, empty buffer, full buffer, etc.).
* This will still hit the same issue if the decorated key size is larger than
expected right? I know almost everyone uses fixed length keys, but it looks
fragile to not at least fail fast if they use variable length keys or if the
size changes in future iterations.
* This is a bug in {{SafeMemoryWriter.length()}}?
* Even if this throws AssertionError it still shouldn't leak memory right? We
should log the error, but not leak. I think (I'll check) that we don't treat
assertion failures as fatal and crash the process.
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145767#comment-16145767
]
Ariel Weisberg commented on CASSANDRA-13785:
Isn't this also a bug in SafeMemoryWriter? Length is supposed to be the amount
of memory written not the size of the allocated buffrer?
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145740#comment-16145740
]
Ariel Weisberg commented on CASSANDRA-13785:
Sure I'm looking at it now. What about the leak? Even if this fails it
shouldn't leak memory right?
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145667#comment-16145667
]
Jay Zhuang commented on CASSANDRA-13785:
[~aweisberg] do you mind reviewing the patch?
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141336#comment-16141336
]
Romain Hardouin commented on CASSANDRA-13785:
-
Thanks!
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140977#comment-16140977
]
Jay Zhuang commented on CASSANDRA-13785:
[~rha] that makes sense. I updated the patch to define the magic number.
Basically, the number is coming from here:
[IndexSummaryBuilder.java:112|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
We potentially could support more entries if the partition key size is smaller
than {{32}}, but I think it's still good to increase the
{{[effectiveMinInterval|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L96]}}
to reduce the summary size automatically in that case.
Also the "large number of keys" more likely have larger partition key size
(like {{32}} or even larger).
| branch | dTest |
| [13785-3.0|https://github.com/cooldoger/cassandra/tree/13785-3.0] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0]
|
| [13785-3.11|https://github.com/cooldoger/cassandra/tree/13785-3.11] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11]
|
| [13785-trunk|https://github.com/cooldoger/cassandra/tree/13785-trunk] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk]
|
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139706#comment-16139706
]
Romain Hardouin commented on CASSANDRA-13785:
-
{code}
private static final int MAX_NUM_ENTRIES = Integer.MAX_VALUE / 40;
{code}
[~jay.zhuang] How did you find {{40}}? Maybe it's wise to name this number with
a constant?
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137614#comment-16137614
]
Jay Zhuang commented on CASSANDRA-13785:
I'm able to reproduce the problem with an unit-test and here is the patch:
| branch | dTest |
| [13785-3.0|https://github.com/cooldoger/cassandra/tree/13785-3.0] |
[circleci#76 passed|https://circleci.com/gh/cooldoger/cassandra/76] |
| [13785-3.11|https://github.com/cooldoger/cassandra/tree/13785-3.11] |
[circleci#77 running|https://circleci.com/gh/cooldoger/cassandra/77] |
| [13785-trunk|https://github.com/cooldoger/cassandra/tree/13785-trunk] |
[circleci#78 running|https://circleci.com/gh/cooldoger/cassandra/78] |
[~iamaleksey] would you please review it?
> Compaction fails for SSTables with large number of keys
> ---
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
