[
https://issues.apache.org/jira/browse/HDDS-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620406#comment-16620406
]
Mukul Kumar Singh commented on HDDS-507:
----------------------------------------
This is happening becaue in unit test, the event handler is being called after
the SCM has been shutdown.
The actual reason for this is because of SCM shutdown, the eventQueue is not
being shutdown.
As part of the fix we should stop all the executors and then shutdown
eventQueue as well.
> RocksDB fails with SEGFAULT randomly during PipelineCloseHandler#onMessage
> --------------------------------------------------------------------------
>
> Key: HDDS-507
> URL: https://issues.apache.org/jira/browse/HDDS-507
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Xiaoyu Yao
> Priority: Major
>
> This can be repro-ed by when TestNodeFailure multiple times. Jenkins
> sometimes also hit this.
>
> {code}
> Current thread (0x00007fbe6f018800): JavaThread
> "EventQueue-PipelineCloseForPipelineCloseHandler" daemon [_thread_in_native,
> id=58639, stack(0x0000700018009000,0x0000700018109000)]
>
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
> 0x000000040000001d
>
>
>
> Stack: [0x0000700018009000,0x0000700018109000], sp=0x0000700018108128, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [librocksdbjni6372054043595793813.jnilib+0x163ac8]
> rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8
> C [librocksdbjni6372054043595793813.jnilib+0x228368]
> rocksdb::DB::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*,
> rocksdb::Slice const&, rocksdb::Slice const&)+0x58
> C [librocksdbjni6372054043595793813.jnilib+0x2282fe]
> rocksdb::DBImpl::Put(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice
> const&)+0xe
> C [librocksdbjni6372054043595793813.jnilib+0x171c84]
> rocksdb::CompactedDBImpl::Open(rocksdb::Options const&,
> std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > const&, rocksdb::DB**)+0x2a4
> C [librocksdbjni6372054043595793813.jnilib+0x971f7]
> rocksdb_put_helper(JNIEnv_*, rocksdb::DB*, rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int, _jbyteArray*, int,
> int)+0x137
> j org.rocksdb.RocksDB.put(JJ[BII[BII)V+0
> j org.rocksdb.RocksDB.put(Lorg/rocksdb/WriteOptions;[B[B)V+17
> j org.apache.hadoop.utils.RocksDBStore.put([B[B)V+10
> j
> org.apache.hadoop.hdds.scm.pipelines.PipelineSelector.updatePipelineState(Lorg/apache/hadoop/hdds/scm/container/common/helpers/Pipeline;Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$LifeCycleEvent;)V+222
> j
> org.apache.hadoop.hdds.scm.pipelines.PipelineSelector.finalizePipeline(Lorg/apache/hadoop/hdds/scm/container/common/helpers/Pipeline;)V+75
> j
> org.apache.hadoop.hdds.scm.container.ContainerMapping.handlePipelineClose(Lorg/apache/hadoop/hdds/scm/container/common/helpers/PipelineID;)V+18
> j
> org.apache.hadoop.hdds.scm.pipelines.PipelineCloseHandler.onMessage(Lorg/apache/hadoop/hdds/scm/container/common/helpers/PipelineID;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+5
> j
> org.apache.hadoop.hdds.scm.pipelines.PipelineCloseHandler.onMessage(Ljava/lang/Object;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+6
> J 5844 C1
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(Lorg/apache/hadoop/hdds/server/events/EventHandler;Ljava/lang/Object;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V
> (41 bytes) @ 0x0000000115c80bc4 [0x0000000115c80aa0+0x124]
> J 5670 C1
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor$$Lambda$143.run()V
> (20 bytes) @ 0x00000001168f625c [0x00000001168f61c0+0x9c]
> j
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
> J 3226 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @
> 0x0000000116356e44 [0x0000000116356d40+0x104]
> J 3107 C1 java.lang.Thread.run()V (17 bytes) @ 0x0000000115d7b0c4
> [0x0000000115d7af80+0x144]
> v ~StubRoutines::call_stub
> V [libjvm.dylib+0x2ef1f6] JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x6ae
> V [libjvm.dylib+0x2ef99a] JavaCalls::call_virtual(JavaValue*, KlassHandle,
> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x164
> V [libjvm.dylib+0x2efb46] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x4a
> V [libjvm.dylib+0x34a46d] thread_entry(JavaThread*, Thread*)+0x7c
> V [libjvm.dylib+0x56eb0f] JavaThread::thread_main_inner()+0x9b
> V [libjvm.dylib+0x57020a] JavaThread::run()+0x1c2
> V [libjvm.dylib+0x48d4a6] java_start(Thread*)+0xf6
> C [libsystem_pthread.dylib+0x3661] _pthread_body+0x154
> C [libsystem_pthread.dylib+0x350d] _pthread_body+0x0
> C [libsystem_pthread.dylib+0x2bf9] thread_start+0xd
> C 0x0000000000000000
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
