[
https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145799#comment-16145799
]
Ariel Weisberg commented on CASSANDRA-13785:
--------------------------------------------
New stuff and summarizing.
* So you are the unlucky person to catch SafeMemoryWriter misbehaving. If we
want our test hygiene to be good you would be the person to write a basic set
of unit tests for it. Go through each external method in SafeMemoryWriter and
test it's API (null handling, largest value, smallest value, negative value, 0)
and internal state (0 length, empty buffer, full buffer, etc.).
* This will still hit the same issue if the decorated key size is larger than
expected right? I know almost everyone uses fixed length keys, but it looks
fragile to not at least fail fast if they use variable length keys or if the
size changes in future iterations.
* This is a bug in {{SafeMemoryWriter.length()}}?
* Even if this throws AssertionError it still shouldn't leak memory right? We
should log the error, but not leak. I think (I'll check) that we don't treat
assertion failures as fatal and crash the process.
> Compaction fails for SSTables with large number of keys
> -------------------------------------------------------
>
> Key: CASSANDRA-13785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Jay Zhuang
> Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
> Memory@[0..d8123468)] was not released before the reference was garbage
> collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the
> compacted SSTable has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will
> fail to create the IndexSummary:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating
> SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
> with {{> Integer.MAX_VALUE}} space, it returns the tailing
> {{Integer.MAX_VALUE}} space
> [SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
> which makes the first
> [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
> not 0. So the assert fails here:
> [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
