[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[
https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116092#comment-15116092
]
Ariel Weisberg commented on CASSANDRA-10688:
I looked at doing this. I think it's more than just a simple stack because I am
trying to bound the memory usage and not just pool. It actually still fails at
that because {{path}} still allocates a large array, but at least it is
proportionally much less than the memory allocated to {{InProgressVisit}}s.
It's a bit finicky to try and clear the stack of things we don't want pooled at
some point while also avoiding walking the entire thing to clear it frequently.
I would like to either remove pooling or leave it as is.
> Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
>
>
> Key: CASSANDRA-10688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10688
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Testing
>Reporter: Jeremiah Jordan
>Assignee: Ariel Weisberg
> Fix For: 3.0.x
>
>
> Running some tests against cassandra-3.0
> 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0
> The tests are just running cassandra-stress write and read while adding and
> removing nodes from the cluster. After the test runs when I go back through
> logs I find the following Stackoverflow fairly often:
> ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638
> Ref.java:413 - Stackoverflow [private java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose,
> final java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen,
> final org.apache.cassandra.cache.InstrumentingCache
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private
> final org.apache.cassandra.cache.ICache
> org.apache.cassandra.cache.InstrumentingCache.map, private final
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final
> com.googlecode.concurrentlinkedhashmap.LinkedDeque
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque,
> com.googlecode.concurrentlinkedhashmap.Linked
> com.googlecode.concurrentlinkedhashmap.LinkedDeque.first,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> ... (repeated a whole bunch more)
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> final java.lang.Object
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key,
> public final byte[] org.apache.cassandra.cache.KeyCacheKey.key
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[
https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15115053#comment-15115053
]
Benedict commented on CASSANDRA-10688:
--
OK, LGTM.
Personally, I would have wrapped the Stack, so that fetch/return of
{{InProgressVisit}} were guaranteed to be paired, and introduced no extra
boilerplate in the main loop (so that the stack pop simply zeroed the top
{{InProgressVisit}} instead of removing it and placing it on another queue).
It's up to you if you want to make that change or not.
> Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
>
>
> Key: CASSANDRA-10688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10688
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Testing
>Reporter: Jeremiah Jordan
>Assignee: Ariel Weisberg
> Fix For: 3.0.x
>
>
> Running some tests against cassandra-3.0
> 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0
> The tests are just running cassandra-stress write and read while adding and
> removing nodes from the cluster. After the test runs when I go back through
> logs I find the following Stackoverflow fairly often:
> ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638
> Ref.java:413 - Stackoverflow [private java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose,
> final java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen,
> final org.apache.cassandra.cache.InstrumentingCache
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private
> final org.apache.cassandra.cache.ICache
> org.apache.cassandra.cache.InstrumentingCache.map, private final
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final
> com.googlecode.concurrentlinkedhashmap.LinkedDeque
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque,
> com.googlecode.concurrentlinkedhashmap.Linked
> com.googlecode.concurrentlinkedhashmap.LinkedDeque.first,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> ... (repeated a whole bunch more)
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> final java.lang.Object
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key,
> public final byte[] org.apache.cassandra.cache.KeyCacheKey.key
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098980#comment-15098980 ] Ariel Weisberg commented on CASSANDRA-10688: OK, looks like it's ready for review again. dtests were failing in Cassci for a while due to config in Cassci, but that seems to have been resolved. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094514#comment-15094514 ] Ariel Weisberg commented on CASSANDRA-10688: It also occurs to me that there are thread safety issues. It's not feasible to iterate collections via iterators because they can be invalidated. I'm going to have to remove that. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094485#comment-15094485 ] Ariel Weisberg commented on CASSANDRA-10688: There is an issue with sun.nio.fs.UnixPath getting stuck. It's iterable and I think it is returning itself. Shouldn't matter because it should be pruned since it is in the visited set, but evidently that isn't happening. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[
https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094531#comment-15094531
]
Benedict commented on CASSANDRA-10688:
--
Darn. True. Instead of removing it you could just special case
{{ConcurrentMap}}, since that's all we care about here. We can in future
special case anything else that trips us up, but it should hopefully be rare we
massively expand the scope of work we need to do.
> Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
>
>
> Key: CASSANDRA-10688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10688
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Testing
>Reporter: Jeremiah Jordan
>Assignee: Ariel Weisberg
> Fix For: 3.0.x
>
>
> Running some tests against cassandra-3.0
> 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0
> The tests are just running cassandra-stress write and read while adding and
> removing nodes from the cluster. After the test runs when I go back through
> logs I find the following Stackoverflow fairly often:
> ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638
> Ref.java:413 - Stackoverflow [private java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose,
> final java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen,
> final org.apache.cassandra.cache.InstrumentingCache
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private
> final org.apache.cassandra.cache.ICache
> org.apache.cassandra.cache.InstrumentingCache.map, private final
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final
> com.googlecode.concurrentlinkedhashmap.LinkedDeque
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque,
> com.googlecode.concurrentlinkedhashmap.Linked
> com.googlecode.concurrentlinkedhashmap.LinkedDeque.first,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> ... (repeated a whole bunch more)
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> final java.lang.Object
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key,
> public final byte[] org.apache.cassandra.cache.KeyCacheKey.key
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092163#comment-15092163 ] Ariel Weisberg commented on CASSANDRA-10688: Rebased and started tests. Also created merge forward branches and tests. |[3.0 code|https://github.com/apache/cassandra/compare/cassandra-3.0...aweisberg:CASSANDRA-10688-3.0?expand=1]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.0-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.0-dtest/]| |[3.3 code|https://github.com/apache/cassandra/compare/cassandra-3.3...aweisberg:CASSANDRA-10688-3.3?expand=1]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.3-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.3-dtest/]| |[trunk code|https://github.com/apache/cassandra/compare/trunk...aweisberg:CASSANDRA-10688-trunk?expand=1]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-trunk-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-trunk-dtest/]| > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node >
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092376#comment-15092376 ] Ariel Weisberg commented on CASSANDRA-10688: Unit tests are timing out. Will look into it. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089602#comment-15089602 ] Michael Shuler commented on CASSANDRA-10688: TE was poking around your branch, and we think CASSANDRA-9303 missing from this dev branch may be causing dtest issues. A re-run is currently in progress, but a 3.0 HEAD rebase might be more fruitful in determining a good comparison with 3.0 HEAD, and then we could re-run another dtest. Just a thought. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089070#comment-15089070 ] Benedict commented on CASSANDRA-10688: -- LGTM; seems to be some issues with dtests though, but they look probably down to flakey environment > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089554#comment-15089554 ] Michael Shuler commented on CASSANDRA-10688: All the dtest jobs are now running on m3.2xlarge instances by default, since a couple weeks ago, which should provide stable dtest results.. let's not just pass off dtests without some due diligence. /me reruns aweisberg-CASSANDRA-10688-3.0-dtest > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088089#comment-15088089 ] Ariel Weisberg commented on CASSANDRA-10688: Updated, tests running now. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[
https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087944#comment-15087944
]
Ariel Weisberg commented on CASSANDRA-10688:
That's not what I am seeing when I run this test case? Am I missing something?
{code}
@Test
public void testArray()
{
int entryCount = 500;
final Object objects[] = new Object[entryCount];
for (int i = 0; i < entryCount; i++)
objects[i] = new Object();
RefCounted.Tidy tidier = new RefCounted.Tidy() {
Object ref = objects;
@Override
public void tidy() throws Exception
{
}
@Override
public String name()
{
return "42";
}
};
Ref ref = new Ref(new AtomicReference(objects),
tidier);
Ref.Visitor visitor = new Ref.Visitor();
visitor.run();
ref.close();
System.out.println(visitor.lastVisitedCount);
System.out.println(visitor.iterations);
Assert.assertTrue(visitor.lastVisitedCount > entryCount &&
visitor.lastVisitedCount < entryCount + 10);
Assert.assertTrue(visitor.iterations > entryCount && visitor.iterations
< entryCount + 10);
}
{code}
I have visitor save the count of the visited set as well as the number of
iterations. It should be quite large since it has to walk all the objects in
the array, but it's skipping the array elements.
> Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
>
>
> Key: CASSANDRA-10688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10688
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Testing
>Reporter: Jeremiah Jordan
>Assignee: Ariel Weisberg
> Fix For: 3.0.x
>
>
> Running some tests against cassandra-3.0
> 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0
> The tests are just running cassandra-stress write and read while adding and
> removing nodes from the cluster. After the test runs when I go back through
> logs I find the following Stackoverflow fairly often:
> ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638
> Ref.java:413 - Stackoverflow [private java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose,
> final java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen,
> final org.apache.cassandra.cache.InstrumentingCache
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private
> final org.apache.cassandra.cache.ICache
> org.apache.cassandra.cache.InstrumentingCache.map, private final
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final
> com.googlecode.concurrentlinkedhashmap.LinkedDeque
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque,
> com.googlecode.concurrentlinkedhashmap.Linked
> com.googlecode.concurrentlinkedhashmap.LinkedDeque.first,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> ... (repeated a whole bunch more)
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
>
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087872#comment-15087872 ] Benedict commented on CASSANDRA-10688: -- Yes, it does > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083025#comment-15083025 ] Benedict commented on CASSANDRA-10688: -- I'm not at all opposed to making the search iterative, but I think a simpler solution (that also improves the running time) is to detect collections/maps and to iterate over them instead of walk the linked-list. This also prevents pollution of the visited set. I'm not sure I would constrain the visited set to as few as 100K items, though, or even 1M. This is a debug feature to ensure we haven't introduced dangerous bugs, and it will simply stop exploring silently and deterministically on certain criteria that might mask those bugs. If we're bounding the visited set we should probably constrain it only to those items in our path, so we no longer save time but still detect loops. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.x > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027886#comment-15027886 ] Jeremiah Jordan commented on CASSANDRA-10688: - Ah. Yes we do have that property set on our tests because we want to get the debug output if there is a leak detected, as these tests have had them happen in the past and just always having that set means we don't have to reproduce it... > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.1, 3.1 > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027758#comment-15027758 ] Ariel Weisberg commented on CASSANDRA-10688: It is just occurring to me, but if we just hit stack overflow what are the odds that logging isn't also going to cause the stack to overflow? I mean in practice it doesn't seem to be happening, but it seems like it should! > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.1, 3.1 > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027803#comment-15027803 ] Ariel Weisberg commented on CASSANDRA-10688: I don't think this is doing what you would hope. I want to rewrite this to be iterative with a fixed bound so that it can abort on depth properly without unexpected behavior. > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.1, 3.1 > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final byte[] org.apache.cassandra.cache.KeyCacheKey.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[ https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027960#comment-15027960 ] Ariel Weisberg commented on CASSANDRA-10688: Proposed change. [~benedict] when you are available you will probably want to review this at some point. The search is iterative and you can set the maximum depth and # of visited objects via system properties. The search records a set of all visited objects so it's a good idea to bound the amount of space that can be used. Right now the maximum depth defaults to 128 and the maximum # of objects visited is 100k. |[3.0 code|https://github.com/apache/cassandra/compare/cassandra-3.0...aweisberg:CASSANDRA-10688-3.0?expand=1]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.0-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10688-3.0-dtest/]| > Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector > > > Key: CASSANDRA-10688 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10688 > Project: Cassandra > Issue Type: Bug >Reporter: Jeremiah Jordan >Assignee: Ariel Weisberg > Fix For: 3.0.1, 3.1 > > > Running some tests against cassandra-3.0 > 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0 > The tests are just running cassandra-stress write and read while adding and > removing nodes from the cluster. After the test runs when I go back through > logs I find the following Stackoverflow fairly often: > ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638 > Ref.java:413 - Stackoverflow [private java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose, > final java.lang.Runnable > org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen, > final org.apache.cassandra.cache.InstrumentingCache > org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private > final org.apache.cassandra.cache.ICache > org.apache.cassandra.cache.InstrumentingCache.map, private final > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap > org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final > com.googlecode.concurrentlinkedhashmap.LinkedDeque > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque, > com.googlecode.concurrentlinkedhashmap.Linked > com.googlecode.concurrentlinkedhashmap.LinkedDeque.first, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > ... (repeated a whole bunch more) > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next, > final java.lang.Object > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.key, > public final
[jira] [Commented] (CASSANDRA-10688) Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
[
https://issues.apache.org/jira/browse/CASSANDRA-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027675#comment-15027675
]
Ariel Weisberg commented on CASSANDRA-10688:
Near as I can tell the stack overflow is being used as a bound for something
that is walking an object graph looking for a path from the outgoing references
of an object to itself doing a depth first search. That isn't a stack trace
it's the graph that it walked (up until it overflowed). I suspect the overflow
is due to the depth of the graph since it's depth first and an any moderately
large linked list is going to overflow pretty quickly.
It's also using Stack which extends Vector which we should probably replace
with ArrayDeque.
This is debug code that only runs if {{-Dcassandra.debugrefcount=true}} so this
isn't an issue in production deployments. [~jjordan] any idea why that would be
set in your experiment?
For debug purposes the code works as designed and it can recover from the stack
overflow and continue searching the graph. It prunes the graph at the point
where the stack overflows. The only real issue is if the error is too noisy.
I think we might want to rate limit it using the first N entries in the graph
as a key. I'll put that together.
> Stack overflow from SSTableReader$InstanceTidier.runOnClose in Leak Detector
>
>
> Key: CASSANDRA-10688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10688
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Ariel Weisberg
> Fix For: 3.0.1, 3.1
>
>
> Running some tests against cassandra-3.0
> 9fc957cf3097e54ccd72e51b2d0650dc3e83eae0
> The tests are just running cassandra-stress write and read while adding and
> removing nodes from the cluster. After the test runs when I go back through
> logs I find the following Stackoverflow fairly often:
> ERROR [Strong-Reference-Leak-Detector:1] 2015-11-11 00:04:10,638
> Ref.java:413 - Stackoverflow [private java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose,
> final java.lang.Runnable
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen,
> final org.apache.cassandra.cache.InstrumentingCache
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache, private
> final org.apache.cassandra.cache.ICache
> org.apache.cassandra.cache.InstrumentingCache.map, private final
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.map, final
> com.googlecode.concurrentlinkedhashmap.LinkedDeque
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.evictionDeque,
> com.googlecode.concurrentlinkedhashmap.Linked
> com.googlecode.concurrentlinkedhashmap.LinkedDeque.first,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> ... (repeated a whole bunch more)
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node.next,
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
