[jira] [Updated] (CASSANDRA-14658) Cassandra hangs at startup

[C. Scott Andreas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

C. Scott Andreas updated CASSANDRA-14658:
-
Component/s: Local/Commit Log

> Cassandra hangs at startup
> --
>
> Key: CASSANDRA-14658
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14658
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Commit Log
>Reporter: Jing Weng
>Priority: Major
> Fix For: 3.11.1
>
>
> Some time ago, when our cassandra cluster started, we found that it could not 
> be started, checked various logs, and found no error message. Later, we 
> obtained the thread stack information at startup by some means, and then 
> refer to the source code, and found that if the commitlog initialization 
> error occurs at startup, cassandra will appear deadlock and hang.
> The thread stack of COMMIT-LOG-ALLOCATOR:
>  
> {noformat}
> {
>   "waitedCount": 0,
>   "lockOwnerId": -1,
>   "lockedMonitors": [],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "RUNNABLE",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "COMMIT-LOG-ALLOCATOR",
>   "lockInfo": null,
>   "threadId": 36,
>   "stackTrace": [
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "runMayThrow",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1",
>   "lineNumber": 133
> },
> {
>   "fileName": "WrappedRunnable.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "org.apache.cassandra.utils.WrappedRunnable",
>   "lineNumber": 28
> },
> {
>   "fileName": "NamedThreadFactory.java",
>   "nativeMethod": false,
>   "methodName": "lambda$threadLocalDeallocator$0",
>   "className": "org.apache.cassandra.concurrent.NamedThreadFactory",
>   "lineNumber": 81
> },
> {
>   "fileName": null,
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": 
> "org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$4\/1392794732",
>   "lineNumber": -1
> },
> {
>   "fileName": "Thread.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "java.lang.Thread",
>   "lineNumber": 748
> }
>   ],
>   "blockedTime": -1,
>   "lockName": null,
>   "lockOwnerName": null,
>   "lockedSynchronizers": []
> }{noformat}
>  
> The thread stack of main thread:
>  
> {noformat}
> {
>   "waitedCount": 2,
>   "lockOwnerId": -1,
>   "lockedMonitors": [
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 10,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 620
>   }
> },
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 11,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 594
>   }
> },
> {
>   "identityHashCode": 1087037934,
>   "lockedStackDepth": 15,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "Keyspace.java",
> "nativeMethod": false,
> "methodName": "open",
> "className": "org.apache.cassandra.db.Keyspace",
> "lineNumber": 127
>   }
> }
>   ],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "WAITING",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "main",
>   "lockInfo": null,
>   "threadId": 1,
>   "stackTrace": [
> {
>   "fileName": "Unsafe.java",
>   "nativeMethod": true,
>   "methodName": "park",
>   "className": "sun.misc.Unsafe",
>   "lineNumber": -2
> },
> {
>   "fileName": "LockSupport.java",
>   "nativeMethod": false,
>   "methodName": "park",
>   "className": "java.util.concurrent.locks.LockSupport",
>   "lineNumber": 304
> },
> {
>   "fileName": "WaitQueue.java",
>   "nativeMethod": false,
>   "methodName": "awaitUninterruptibly",
>   "className": 
> "org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal",
>   "lineNumber": 280
> },
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "awaitAvailableSegment",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager",
>   

[jira] [Updated] (CASSANDRA-14658) Cassandra hangs at startup

[C. Scott Andreas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

C. Scott Andreas updated CASSANDRA-14658:
-
Reproduced In: 3.11.1

> Cassandra hangs at startup
> --
>
> Key: CASSANDRA-14658
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14658
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Commit Log
>Reporter: Jing Weng
>Priority: Major
>
> Some time ago, when our cassandra cluster started, we found that it could not 
> be started, checked various logs, and found no error message. Later, we 
> obtained the thread stack information at startup by some means, and then 
> refer to the source code, and found that if the commitlog initialization 
> error occurs at startup, cassandra will appear deadlock and hang.
> The thread stack of COMMIT-LOG-ALLOCATOR:
>  
> {noformat}
> {
>   "waitedCount": 0,
>   "lockOwnerId": -1,
>   "lockedMonitors": [],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "RUNNABLE",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "COMMIT-LOG-ALLOCATOR",
>   "lockInfo": null,
>   "threadId": 36,
>   "stackTrace": [
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "runMayThrow",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1",
>   "lineNumber": 133
> },
> {
>   "fileName": "WrappedRunnable.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "org.apache.cassandra.utils.WrappedRunnable",
>   "lineNumber": 28
> },
> {
>   "fileName": "NamedThreadFactory.java",
>   "nativeMethod": false,
>   "methodName": "lambda$threadLocalDeallocator$0",
>   "className": "org.apache.cassandra.concurrent.NamedThreadFactory",
>   "lineNumber": 81
> },
> {
>   "fileName": null,
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": 
> "org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$4\/1392794732",
>   "lineNumber": -1
> },
> {
>   "fileName": "Thread.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "java.lang.Thread",
>   "lineNumber": 748
> }
>   ],
>   "blockedTime": -1,
>   "lockName": null,
>   "lockOwnerName": null,
>   "lockedSynchronizers": []
> }{noformat}
>  
> The thread stack of main thread:
>  
> {noformat}
> {
>   "waitedCount": 2,
>   "lockOwnerId": -1,
>   "lockedMonitors": [
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 10,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 620
>   }
> },
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 11,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 594
>   }
> },
> {
>   "identityHashCode": 1087037934,
>   "lockedStackDepth": 15,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "Keyspace.java",
> "nativeMethod": false,
> "methodName": "open",
> "className": "org.apache.cassandra.db.Keyspace",
> "lineNumber": 127
>   }
> }
>   ],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "WAITING",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "main",
>   "lockInfo": null,
>   "threadId": 1,
>   "stackTrace": [
> {
>   "fileName": "Unsafe.java",
>   "nativeMethod": true,
>   "methodName": "park",
>   "className": "sun.misc.Unsafe",
>   "lineNumber": -2
> },
> {
>   "fileName": "LockSupport.java",
>   "nativeMethod": false,
>   "methodName": "park",
>   "className": "java.util.concurrent.locks.LockSupport",
>   "lineNumber": 304
> },
> {
>   "fileName": "WaitQueue.java",
>   "nativeMethod": false,
>   "methodName": "awaitUninterruptibly",
>   "className": 
> "org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal",
>   "lineNumber": 280
> },
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "awaitAvailableSegment",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager",
>   "lineNumber": 262
> },
> {
>

[jira] [Updated] (CASSANDRA-14658) Cassandra hangs at startup

[C. Scott Andreas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

C. Scott Andreas updated CASSANDRA-14658:
-
Fix Version/s: (was: 3.11.1)

> Cassandra hangs at startup
> --
>
> Key: CASSANDRA-14658
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14658
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Commit Log
>Reporter: Jing Weng
>Priority: Major
>
> Some time ago, when our cassandra cluster started, we found that it could not 
> be started, checked various logs, and found no error message. Later, we 
> obtained the thread stack information at startup by some means, and then 
> refer to the source code, and found that if the commitlog initialization 
> error occurs at startup, cassandra will appear deadlock and hang.
> The thread stack of COMMIT-LOG-ALLOCATOR:
>  
> {noformat}
> {
>   "waitedCount": 0,
>   "lockOwnerId": -1,
>   "lockedMonitors": [],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "RUNNABLE",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "COMMIT-LOG-ALLOCATOR",
>   "lockInfo": null,
>   "threadId": 36,
>   "stackTrace": [
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "runMayThrow",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1",
>   "lineNumber": 133
> },
> {
>   "fileName": "WrappedRunnable.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "org.apache.cassandra.utils.WrappedRunnable",
>   "lineNumber": 28
> },
> {
>   "fileName": "NamedThreadFactory.java",
>   "nativeMethod": false,
>   "methodName": "lambda$threadLocalDeallocator$0",
>   "className": "org.apache.cassandra.concurrent.NamedThreadFactory",
>   "lineNumber": 81
> },
> {
>   "fileName": null,
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": 
> "org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$4\/1392794732",
>   "lineNumber": -1
> },
> {
>   "fileName": "Thread.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "java.lang.Thread",
>   "lineNumber": 748
> }
>   ],
>   "blockedTime": -1,
>   "lockName": null,
>   "lockOwnerName": null,
>   "lockedSynchronizers": []
> }{noformat}
>  
> The thread stack of main thread:
>  
> {noformat}
> {
>   "waitedCount": 2,
>   "lockOwnerId": -1,
>   "lockedMonitors": [
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 10,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 620
>   }
> },
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 11,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 594
>   }
> },
> {
>   "identityHashCode": 1087037934,
>   "lockedStackDepth": 15,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "Keyspace.java",
> "nativeMethod": false,
> "methodName": "open",
> "className": "org.apache.cassandra.db.Keyspace",
> "lineNumber": 127
>   }
> }
>   ],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "WAITING",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "main",
>   "lockInfo": null,
>   "threadId": 1,
>   "stackTrace": [
> {
>   "fileName": "Unsafe.java",
>   "nativeMethod": true,
>   "methodName": "park",
>   "className": "sun.misc.Unsafe",
>   "lineNumber": -2
> },
> {
>   "fileName": "LockSupport.java",
>   "nativeMethod": false,
>   "methodName": "park",
>   "className": "java.util.concurrent.locks.LockSupport",
>   "lineNumber": 304
> },
> {
>   "fileName": "WaitQueue.java",
>   "nativeMethod": false,
>   "methodName": "awaitUninterruptibly",
>   "className": 
> "org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal",
>   "lineNumber": 280
> },
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "awaitAvailableSegment",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager",
>   "lineNumber": 262
> },
> 

[jira] [Updated] (CASSANDRA-14658) Cassandra hangs at startup

[C. Scott Andreas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

C. Scott Andreas updated CASSANDRA-14658:
-
Component/s: (was: Legacy/Core)

> Cassandra hangs at startup
> --
>
> Key: CASSANDRA-14658
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14658
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jing Weng
>Priority: Major
> Fix For: 3.11.1
>
>
> Some time ago, when our cassandra cluster started, we found that it could not 
> be started, checked various logs, and found no error message. Later, we 
> obtained the thread stack information at startup by some means, and then 
> refer to the source code, and found that if the commitlog initialization 
> error occurs at startup, cassandra will appear deadlock and hang.
> The thread stack of COMMIT-LOG-ALLOCATOR:
>  
> {noformat}
> {
>   "waitedCount": 0,
>   "lockOwnerId": -1,
>   "lockedMonitors": [],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "RUNNABLE",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "COMMIT-LOG-ALLOCATOR",
>   "lockInfo": null,
>   "threadId": 36,
>   "stackTrace": [
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "runMayThrow",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1",
>   "lineNumber": 133
> },
> {
>   "fileName": "WrappedRunnable.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "org.apache.cassandra.utils.WrappedRunnable",
>   "lineNumber": 28
> },
> {
>   "fileName": "NamedThreadFactory.java",
>   "nativeMethod": false,
>   "methodName": "lambda$threadLocalDeallocator$0",
>   "className": "org.apache.cassandra.concurrent.NamedThreadFactory",
>   "lineNumber": 81
> },
> {
>   "fileName": null,
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": 
> "org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$4\/1392794732",
>   "lineNumber": -1
> },
> {
>   "fileName": "Thread.java",
>   "nativeMethod": false,
>   "methodName": "run",
>   "className": "java.lang.Thread",
>   "lineNumber": 748
> }
>   ],
>   "blockedTime": -1,
>   "lockName": null,
>   "lockOwnerName": null,
>   "lockedSynchronizers": []
> }{noformat}
>  
> The thread stack of main thread:
>  
> {noformat}
> {
>   "waitedCount": 2,
>   "lockOwnerId": -1,
>   "lockedMonitors": [
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 10,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 620
>   }
> },
> {
>   "identityHashCode": 600118828,
>   "lockedStackDepth": 11,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "ColumnFamilyStore.java",
> "nativeMethod": false,
> "methodName": "createColumnFamilyStore",
> "className": "org.apache.cassandra.db.ColumnFamilyStore",
> "lineNumber": 594
>   }
> },
> {
>   "identityHashCode": 1087037934,
>   "lockedStackDepth": 15,
>   "className": "java.lang.Class",
>   "lockedStackFrame": {
> "fileName": "Keyspace.java",
> "nativeMethod": false,
> "methodName": "open",
> "className": "org.apache.cassandra.db.Keyspace",
> "lineNumber": 127
>   }
> }
>   ],
>   "waitedTime": -1,
>   "blockedCount": 0,
>   "threadState": "WAITING",
>   "inNative": false,
>   "suspended": false,
>   "threadName": "main",
>   "lockInfo": null,
>   "threadId": 1,
>   "stackTrace": [
> {
>   "fileName": "Unsafe.java",
>   "nativeMethod": true,
>   "methodName": "park",
>   "className": "sun.misc.Unsafe",
>   "lineNumber": -2
> },
> {
>   "fileName": "LockSupport.java",
>   "nativeMethod": false,
>   "methodName": "park",
>   "className": "java.util.concurrent.locks.LockSupport",
>   "lineNumber": 304
> },
> {
>   "fileName": "WaitQueue.java",
>   "nativeMethod": false,
>   "methodName": "awaitUninterruptibly",
>   "className": 
> "org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal",
>   "lineNumber": 280
> },
> {
>   "fileName": "AbstractCommitLogSegmentManager.java",
>   "nativeMethod": false,
>   "methodName": "awaitAvailableSegment",
>   "className": 
> "org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager",
>   "lineNumber": 262
> },
>