[
https://issues.apache.org/jira/browse/CASSANDRA-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749758#action_12749758
]
Sammy Yu edited comment on CASSANDRA-405 at 8/31/09 9:59 PM:
-------------------------------------------------------------
We did multiple stack dumps they are all the same. There is no progression.
In this stack trace there are two sets of stack traces
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable
[0x0000000046532000]
java.lang.Thread.State: RUNNABLE
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at
org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator.
(SSTableNamesIterator.java:46)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable
[0x0000000046330000]
java.lang.Thread.State: RUNNABLE
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at
org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator.
(SSTableNamesIterator.java:46)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
All the other pool-1-threads are in the readLock waiting state due to the
writeLock
"pool-1-thread-12425" prio=10 tid=0x00007f1b7857e000 nid=0x3fe6 waiting on
condition [0x00007f1892528000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00007f1b8534e848> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:877)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1412)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:609)
at
org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:320)
at
org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
at
org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:173)
at
org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:213)
at
org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:551)
at
org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:539)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
was (Author: sammy.yu):
In this stack trace there are two sets of threads that are stuck in the
iterator
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable
[0x0000000046532000]
java.lang.Thread.State: RUNNABLE
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at
org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator.
(SSTableNamesIterator.java:46)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable
[0x0000000046330000]
java.lang.Thread.State: RUNNABLE
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at
com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at
org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator.
(SSTableNamesIterator.java:46)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
> Race condition with ConcurrentLinkedHashMap
> -------------------------------------------
>
> Key: CASSANDRA-405
> URL: https://issues.apache.org/jira/browse/CASSANDRA-405
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Chris Goffinet
> Attachments: stack.log.gz
>
>
> We are seeing a race condition with ConcurrentLinkedHashMap using
> appendToTail. We could remove the ConcurrentLinkedHashMap for now until
> that's resolved.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
