Jon Meredith created CASSANDRA-21855:
----------------------------------------
Summary: CMS node with the lowest NodeId can deadlock on
nonPeriodicTasks when it replays a delayed log that triggers a snapshot and
then drops a table
Key: CASSANDRA-21855
URL: https://issues.apache.org/jira/browse/CASSANDRA-21855
Project: Apache Cassandra
Issue Type: Bug
Components: Transactional Cluster Metadata
Reporter: Jon Meredith
Assignee: Jon Meredith
The CMS member with the lowest {{NodeId}} can stop processing the TCM log until
restarted. It happens when the node is behind, enacts an epoch that triggers a
metadata snapshot, and then enacts a table drop. The snapshot task and the drop
wait on each other through {{ScheduledExecutors.nonPeriodicTasks}}, and neither
wait has a timeout. The node needs a restart.
The rest of the cluster keeps moving. On the stuck node, {{nodetool cms
describe}} shows a stale local epoch and a growing count of pending entries.
h3. Cause
* {{LocalLog.snapshotListener()}} fires when an enacted epoch is a multiple of
{{metadata_snapshot_frequency}} and the local node has the lowest {{NodeId}}
among the full CMS members. It submitted
{{ClusterMetadataService.triggerSnapshot()}} to
{{{}ScheduledExecutors.nonPeriodicTasks{}}}.
* {{nonPeriodicTasks}} has one thread. {{LogTransaction.waitForDeletions}}
uses it as a barrier. It schedules a no-op on the executor and blocks until the
queue drains past it.
* The log follower enacts schema changes inline. For a dropped table it
reaches the barrier through {{{}SchemaListener.notifyPreCommit{}}},
{{{}DistributedSchema.initializeKeyspaceInstances{}}}, {{{}Keyspace.dropCf{}}},
{{ColumnFamilyStore.invalidate}} and
{{{}LifecycleTransaction.waitForDeletions{}}}.
* {{ClusterMetadataService.commit}} waits at its pre-commit barrier
({{{}LocalLog.waitForHighestConsecutive{}}}) without a deadline, before it
consults its retry policy. If the snapshot task already holds the executor's
thread, it parks there until the follower finishes a loop.
* The follower is blocked in the deletion barrier and needs that thread to run
its no-op. The wait is circular. Every later commit on the node also parks at
the same barrier.
h3. Stack traces
The snapshot task holding the only {{nonPeriodicTasks}} thread and waiting on
the follower.
{code}
"NonPeriodicTasks:1" WAITING (parking)
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await
at org.apache.cassandra.tcm.log.LocalLog$Async.runOnce
at org.apache.cassandra.tcm.log.LocalLog.runOnce
at org.apache.cassandra.tcm.log.LocalLog.waitForHighestConsecutive
at org.apache.cassandra.tcm.ClusterMetadataService.commit
at org.apache.cassandra.tcm.ClusterMetadataService.commit
at org.apache.cassandra.tcm.ClusterMetadataService.triggerSnapshot
at org.apache.cassandra.tcm.log.LocalLog.lambda$snapshotListener$3
at org.apache.cassandra.concurrent.FutureTask.run
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
{code}
The log follower enacting a dropped table and waiting on the same executor.
{code}
"GlobalLogFollower" WAITING (parking)
- parking to wait for a
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
at java.util.concurrent.FutureTask.get
at org.apache.cassandra.utils.FBUtilities.waitOnFuture
at org.apache.cassandra.db.lifecycle.LogTransaction.waitForDeletions
at org.apache.cassandra.db.lifecycle.LifecycleTransaction.waitForDeletions
at org.apache.cassandra.db.ColumnFamilyStore.invalidate
at org.apache.cassandra.db.Keyspace.unloadCf
at org.apache.cassandra.db.Keyspace.dropCf
at org.apache.cassandra.schema.DistributedSchema.dropTable
at org.apache.cassandra.schema.DistributedSchema.initializeKeyspaceInstances
at org.apache.cassandra.tcm.listeners.SchemaListener.notifyPreCommit
at org.apache.cassandra.tcm.log.LocalLog.notifyPreCommit
at org.apache.cassandra.tcm.log.LocalLog.processPendingInternal
at org.apache.cassandra.tcm.log.LocalLog$Async$AsyncRunnable.run}}
{code}
The park address in the second trace is the future for the no-op that the first
thread prevents from running.
h3. Fix
Trigger snapshots from a private single-thread executor, {{CMSLogTasks}}, owned
by {{ClusterMetadataService}}. {{nonPeriodicTasks}} then no longer carries the
snapshot task, so the deletion barrier can drain. The executor is shut down in
{{StorageService.drain()}} and in the in-JVM dtest instance teardown.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
