Zephyr Guo created CASSANDRA-15295:
--------------------------------------
Summary: Running into deadlock when do CommitLog initialization
Key: CASSANDRA-15295
URL: https://issues.apache.org/jira/browse/CASSANDRA-15295
Project: Cassandra
Issue Type: Bug
Components: Local/Commit Log
Reporter: Zephyr Guo
Assignee: Zephyr Guo
Attachments: image-2019-08-30-21-25-17-638.png,
image-2019-08-30-21-29-48-623.png, image-2019-08-30-21-30-11-683.png,
image-2019-08-30-21-40-43-748.png
Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long
time.
I used jstack to saw what happened. The main thread stuck in
*AbstractCommitLogSegmentManager.awaitAvailableSegment*
!image-2019-08-30-21-40-43-748.png|thumbnail!
The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was
not actually running.
!image-2019-08-30-21-30-11-683.png|thumbnail!
And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on
java class initialization.
!image-2019-08-30-21-25-17-638.png|thumbnail!
This is a deadlock obviously. CommitLog waits for a CommitLogSegment when
initializing. In this moment, the CommitLog class is not initialized and the
main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a
CommitLogSegment with exception and call *CommitLog.handleCommitError*(static
method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class
is still initializing.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
