[
https://issues.apache.org/jira/browse/CASSANDRA-21855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Meredith updated CASSANDRA-21855:
-------------------------------------
Fix Version/s: 6.0-alpha
(was: 6.0)
> CMS node with the lowest NodeId can deadlock on nonPeriodicTasks when it
> replays a delayed log that triggers a snapshot and then drops a table
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21855
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21855
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Transactional Cluster Metadata
> Reporter: Jon Meredith
> Assignee: Jon Meredith
> Priority: Normal
> Fix For: 6.0-alpha, 7.x
>
>
> The CMS member with the lowest {{NodeId}} can stop processing the TCM log
> until restarted. It happens when the node is behind, enacts an epoch that
> triggers a metadata snapshot, and then enacts a table drop. The snapshot task
> and the drop wait on each other through
> {{ScheduledExecutors.nonPeriodicTasks}}, and neither wait has a timeout. The
> node needs a restart.
> The rest of the cluster keeps moving. On the stuck node, {{nodetool cms
> describe}} shows a stale local epoch and a growing count of pending entries.
> h3. Cause
> * {{LocalLog.snapshotListener()}} fires when an enacted epoch is a multiple
> of {{metadata_snapshot_frequency}} and the local node has the lowest
> {{NodeId}} among the full CMS members. It submitted
> {{ClusterMetadataService.triggerSnapshot()}} to
> {{{}ScheduledExecutors.nonPeriodicTasks{}}}.
> * {{nonPeriodicTasks}} has one thread. {{LogTransaction.waitForDeletions}}
> uses it as a barrier. It schedules a no-op on the executor and blocks until
> the queue drains past it.
> * The log follower enacts schema changes inline. For a dropped table it
> reaches the barrier through {{{}SchemaListener.notifyPreCommit{}}},
> {{{}DistributedSchema.initializeKeyspaceInstances{}}},
> {{{}Keyspace.dropCf{}}}, {{ColumnFamilyStore.invalidate}} and
> {{{}LifecycleTransaction.waitForDeletions{}}}.
> * {{ClusterMetadataService.commit}} waits at its pre-commit barrier
> ({{{}LocalLog.waitForHighestConsecutive{}}}) without a deadline, before it
> consults its retry policy. If the snapshot task already holds the executor's
> thread, it parks there until the follower finishes a loop.
> * The follower is blocked in the deletion barrier and needs that thread to
> run its no-op. The wait is circular. Every later commit on the node also
> parks at the same barrier.
> h3. Stack traces
> The snapshot task holding the only {{nonPeriodicTasks}} thread and waiting on
> the follower.
> {code}
> "NonPeriodicTasks:1" WAITING (parking)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await
> at org.apache.cassandra.tcm.log.LocalLog$Async.runOnce
> at org.apache.cassandra.tcm.log.LocalLog.runOnce
> at org.apache.cassandra.tcm.log.LocalLog.waitForHighestConsecutive
> at org.apache.cassandra.tcm.ClusterMetadataService.commit
> at org.apache.cassandra.tcm.ClusterMetadataService.commit
> at org.apache.cassandra.tcm.ClusterMetadataService.triggerSnapshot
> at org.apache.cassandra.tcm.log.LocalLog.lambda$snapshotListener$3
> at org.apache.cassandra.concurrent.FutureTask.run
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
> {code}
> The log follower enacting a dropped table and waiting on the same executor.
> {code}
> "GlobalLogFollower" WAITING (parking)
> - parking to wait for a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
> at java.util.concurrent.FutureTask.get
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture
> at org.apache.cassandra.db.lifecycle.LogTransaction.waitForDeletions
> at org.apache.cassandra.db.lifecycle.LifecycleTransaction.waitForDeletions
> at org.apache.cassandra.db.ColumnFamilyStore.invalidate
> at org.apache.cassandra.db.Keyspace.unloadCf
> at org.apache.cassandra.db.Keyspace.dropCf
> at org.apache.cassandra.schema.DistributedSchema.dropTable
> at
> org.apache.cassandra.schema.DistributedSchema.initializeKeyspaceInstances
> at org.apache.cassandra.tcm.listeners.SchemaListener.notifyPreCommit
> at org.apache.cassandra.tcm.log.LocalLog.notifyPreCommit
> at org.apache.cassandra.tcm.log.LocalLog.processPendingInternal
> at org.apache.cassandra.tcm.log.LocalLog$Async$AsyncRunnable.run}}
> {code}
> The park address in the second trace is the future for the no-op that the
> first thread prevents from running.
> h3. Fix
> Trigger snapshots from a private single-thread executor, {{CMSLogTasks}},
> owned by {{ClusterMetadataService}}. {{nonPeriodicTasks}} then no longer
> carries the snapshot task, so the deletion barrier can drain. The executor is
> shut down in {{StorageService.drain()}} and in the in-JVM dtest instance
> teardown.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
