[
https://issues.apache.org/jira/browse/CASSANDRA-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Meredith updated CASSANDRA-17524:
-------------------------------------
Status: Ready to Commit (was: Review In Progress)
Starting commit
CI Results (pending):
||Branch||Source||Circle CI||Jenkins||
|cassandra-3.0|[branch|https://github.com/jonmeredith/cassandra/tree/commit_remote_branch/CASSANDRA-17524-cassandra-3.0-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://app.circleci.com/pipelines/github/jonmeredith/cassandra?branch=commit_remote_branch%2FCASSANDRA-17524-cassandra-3.0-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/1632/]|
|cassandra-3.11|[branch|https://github.com/jonmeredith/cassandra/tree/commit_remote_branch/CASSANDRA-17524-cassandra-3.11-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://app.circleci.com/pipelines/github/jonmeredith/cassandra?branch=commit_remote_branch%2FCASSANDRA-17524-cassandra-3.11-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/1633/]|
|cassandra-4.0|[branch|https://github.com/jonmeredith/cassandra/tree/commit_remote_branch/CASSANDRA-17524-cassandra-4.0-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://app.circleci.com/pipelines/github/jonmeredith/cassandra?branch=commit_remote_branch%2FCASSANDRA-17524-cassandra-4.0-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/1634/]|
|trunk|[branch|https://github.com/jonmeredith/cassandra/tree/commit_remote_branch/CASSANDRA-17524-trunk-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://app.circleci.com/pipelines/github/jonmeredith/cassandra?branch=commit_remote_branch%2FCASSANDRA-17524-trunk-B2CD88DD-CD43-41E1-AC67-692E1D870936]|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/1635/]|
> Schema mutations may not be completed on drain
> ----------------------------------------------
>
> Key: CASSANDRA-17524
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17524
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: Jon Meredith
> Assignee: Jon Meredith
> Priority: Normal
> Fix For: 4.1, 3.0.x, 3.11.x, 4.0.x
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The drain logic (invoked explicitly with nodetool or from the JVM
> shutdown hook) closes down executor stages that can create mutations (counter,
> view, mutation) before closing down the commitlog. The gossip
> stage also commits schema mutations, and should be treated the same way.
> The messaging service is shut down as part of drain, so there should be
> no new Gossip messages received, however any messages still queued
> in the executor could still run after the commitlog allocator is shut down as
> part of drain, causing the gossip stage thread to hang indefinitely waiting
> for a new segment that never arrives.
> Here is an example from an in-JVM dtest, showing an update to the peers table
> as it shuts down.
> {code:java}
> park:-1, Unsafe (jdk.internal.misc)
> park:323, LockSupport (java.util.concurrent.locks)
> await:289, WaitQueue$Standard$AbstractSignal
> (org.apache.cassandra.utils.concurrent)
> await:282, WaitQueue$Standard$AbstractSignal
> (org.apache.cassandra.utils.concurrent)
> awaitUninterruptibly:186, Awaitable$Defaults
> (org.apache.cassandra.utils.concurrent)
> awaitUninterruptibly:259, Awaitable$AbstractAwaitable
> (org.apache.cassandra.utils.concurrent)
> awaitAvailableSegment:283, AbstractCommitLogSegmentManager
> (org.apache.cassandra.db.commitlog)
> advanceAllocatingFrom:257, AbstractCommitLogSegmentManager
> (org.apache.cassandra.db.commitlog)
> allocate:55, CommitLogSegmentManagerStandard
> (org.apache.cassandra.db.commitlog)
> add:282, CommitLog (org.apache.cassandra.db.commitlog)
> beginWrite:50, CassandraKeyspaceWriteHandler (org.apache.cassandra.db)
> applyInternal:622, Keyspace (org.apache.cassandra.db)
> apply:506, Keyspace (org.apache.cassandra.db)
> apply:215, Mutation (org.apache.cassandra.db)
> apply:220, Mutation (org.apache.cassandra.db)
> apply:229, Mutation (org.apache.cassandra.db)
> executeInternalWithoutCondition:644, ModificationStatement
> (org.apache.cassandra.cql3.statements)
> executeLocally:635, ModificationStatement
> (org.apache.cassandra.cql3.statements)
> executeInternal:431, QueryProcessor (org.apache.cassandra.cql3)
> updateTokens:804, SystemKeyspace (org.apache.cassandra.db)
> updateTokenMetadata:2941, StorageService (org.apache.cassandra.service)
> handleStateNormal:3057, StorageService (org.apache.cassandra.service)
> onChange:2498, StorageService (org.apache.cassandra.service)
> markAsShutdown:607, Gossiper (org.apache.cassandra.gms)
> doVerb:39, GossipShutdownVerbHandler (org.apache.cassandra.gms)
> lambda$new$0:78, InboundSink (org.apache.cassandra.net)
> accept:-1, 581110313 (org.apache.cassandra.net.InboundSink$$Lambda$2638)
> accept:64, InboundSink$Filtered (org.apache.cassandra.net)
> accept:50, InboundSink$Filtered (org.apache.cassandra.net)
> accept:97, InboundSink (org.apache.cassandra.net)
> accept:45, InboundSink (org.apache.cassandra.net)
> run:433, InboundMessageHandler$ProcessMessage (org.apache.cassandra.net)
> run:124, ExecutionFailure$1 (org.apache.cassandra.concurrent)
> runWorker:1128, ThreadPoolExecutor (java.util.concurrent)
> run:628, ThreadPoolExecutor$Worker (java.util.concurrent)
> run:30, FastThreadLocalRunnable (io.netty.util.concurrent)
> run:829, Thread (java.lang)
> {code}
> This causes an exception during shutdown for the in-JVM dtest as it is
> unable to shutdown {{{}Stage.GOSSIP{}}}, but does not prevent regular
> shutdown for Cassandra as the executors are not stopped. The schema update
> would be lost, despite requesting a graceful shutdown.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
