Tyler Hobbs created CASSANDRA-10181:
---------------------------------------
Summary: Deadlock flushing tables with CUSTOM indexes
Key: CASSANDRA-10181
URL: https://issues.apache.org/jira/browse/CASSANDRA-10181
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Tyler Hobbs
Assignee: Sam Tunnicliffe
Fix For: 3.0 beta 2
Attachments: flush-deadlock-repro.txt
In 3.0, if a table with a CUSTOM secondary index is force flushed, Cassandra
will deadlock while attempting to perform a blocking flush on the tables
backing the secondary indexes.
The basic problem is that the base table's post-flush task ends up waiting on
the post-flush task for the secondary index to complete. However, since the
post-flush executor is single-threaded, this results in a deadlock.
Here's the partial stacktrace for the base table part of this (line numbers may
not be 100% accurate):
{noformat}
org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:927)
at
org.apache.cassandra.index.internal.CustomIndex.lambda$getBlockingFlushTask$0(VertexCentricIndex.java:114)
at
org.apache.cassandra.index.internal.CustomIndex$$Lambda$95/057902870.call(Unknown
Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
at
org.apache.cassandra.index.SecondaryIndexManager.lambda$executeAllBlocking$39(SecondaryIndexManager.java:896)
at
org.apache.cassandra.index.SecondaryIndexManager$$Lambda$94/25774682.accept(Unknown
Source)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at
org.apache.cassandra.index.SecondaryIndexManager.executeAllBlocking(SecondaryIndexManager.java:893)
at
org.apache.cassandra.index.SecondaryIndexManager.flushIndexesBlocking(SecondaryIndexManager.java:346)
at
org.apache.cassandra.index.SecondaryIndexManager.flushAllCustomIndexesBlocking(SecondaryIndexManager.java:358)
at
org.apache.cassandra.db.ColumnFamilyStore$PostFlush.run(ColumnFamilyStore.java:960)
{noformat}
First, note that the base of this stacktrace is in CFS$PostFlush.run(), which
means it's running on the post-flush executor. When
{{CFS.forceBlockingFlush()}} is called on the secondary index table, we end up
blocking on another task that's submitted to the post-flush executor. Since
that executor is single-threaded and is already running the base table task,
this results in deadlock.
The attached patch includes a unit test and custom secondary index class
(basically just KeysIndex) to reproduce the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
