nandakumar131 commented on pull request #1019: URL: https://github.com/apache/hadoop-ozone/pull/1019#issuecomment-650600652
The crash is not due to RocksDB bug. We are trying to access the DB after closing it, which is causing the crash. The `BackgroundPipelineCreator` thread is trying to access RocksDB after the DB is closed using junit's `@After` method. ```C [librocksdbjni5084871938501755619.jnilib+0x102485] _ZN7rocksdb2DB3PutERKNS_12WriteOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceES8_+0x35 C [librocksdbjni5084871938501755619.jnilib+0x10243e] _ZN7rocksdb6DBImpl3PutERKNS_12WriteOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceES8_+0xe C [librocksdbjni5084871938501755619.jnilib+0x181b1] _Z18rocksdb_put_helperP7JNIEnv_PN7rocksdb2DBERKNS1_12WriteOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayiiSA_ii+0x131 j org.rocksdb.RocksDB.put(JJ[BII[BIIJ)V+0 j org.rocksdb.RocksDB.put(Lorg/rocksdb/ColumnFamilyHandle;Lorg/rocksdb/WriteOptions;[B[B)V+23 j org.apache.hadoop.hdds.utils.db.RDBTable.put([B[B)V+14 j org.apache.hadoop.hdds.utils.db.RDBTable.put(Ljava/lang/Object;Ljava/lang/Object;)V+9 j org.apache.hadoop.hdds.utils.db.TypedTable.put(Ljava/lang/Object;Ljava/lang/Object;)V+26 j org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.addPipeline(Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$Pipeline;)V+14 j sun.reflect.GeneratedMethodAccessor7.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+40 J 1739 C2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x000000010a9f7ba8 [0x000000010a9f7b00+0xa8] j org.apache.hadoop.hdds.scm.ha.MockSCMHAManager$MockRatisServer.process(Lorg/apache/hadoop/hdds/scm/ha/SCMRatisRequest;)Lorg/apache/ratis/protocol/Message;+131 j org.apache.hadoop.hdds.scm.ha.MockSCMHAManager$MockRatisServer.submitRequest(Lorg/apache/hadoop/hdds/scm/ha/SCMRatisRequest;)Lorg/apache/hadoop/hdds/scm/ha/SCMRatisResponse;+14 j org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatis(Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+31 J 1944 C1 org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x000000010aa7728c [0x000000010aa76c40+0x64c] j com.sun.proxy.$Proxy9.addPipeline(Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$Pipeline;)V+16 j org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.createPipeline(Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$ReplicationType;Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$ReplicationFactor;)Lorg/apache/hadoop/hdds/scm/pipeline/Pipeline;+66 j org.apache.hadoop.hdds.scm.pipeline.BackgroundPipelineCreator.createPipelines()V+130 j org.apache.hadoop.hdds.scm.pipeline.BackgroundPipelineCreator$$Lambda$20.run()V+4 j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 j java.util.concurrent.FutureTask.run()V+42 j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)V+1 j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V+30 j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 ``` Ideally, the test case execution should end when `@After` method is executed. In our case, the `@After` method and `BackgroundPipelineCreator` are running at the same time causing intermittent JVM crash. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
