LuciferYang commented on code in PR #37610:
URL: https://github.com/apache/spark/pull/37610#discussion_r951185256
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -802,6 +802,7 @@ public void close() {
if (db != null) {
try {
db.close();
+ db = null;
Review Comment:
@zhouyejoe should we set `db = null`?
I found that `Finalized merged shuffle are written into DB and cleaned up
after application stopped` in `YarnShuffleServiceWithRocksDBBackendSuite` will
crash when use RocksDB.
The native stack as follows:
```
Stack: [0x0000000288410000,0x0000000288813000], sp=0x0000000288812150,
free space=4104k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
C [librocksdbjni6224075543374471149.jnilib+0xd4ec8]
_ZN7rocksdb15CompactedDBImpl3GetERKNS_11ReadOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceEPNS_13PinnableSliceEPNSt3__112basic_stringIcNSB_11char_traitsIcEENSB_9allocatorIcEEEE+0xc6c
C [librocksdbjni6224075543374471149.jnilib+0x1406a0]
_ZN7rocksdb6DBImpl6DeleteERKNS_12WriteOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceE+0x38
C [librocksdbjni6224075543374471149.jnilib+0x1e38c]
_Z21rocksdb_delete_helperP7JNIEnv_PN7rocksdb2DBERKNS1_12WriteOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xe0
j org.rocksdb.RocksDB.delete(J[BII)V+0
j org.rocksdb.RocksDB.delete([B)V+9
j org.apache.spark.network.shuffledb.RocksDB.delete([B)V+5
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShufflePartitionInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppAttemptShuffleMergeId;)V+16
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$removeAppShuffleInfoFromDB$8(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Ljava/lang/Integer;Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleMergePartitionsInfo;)V+24
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$321.accept(Ljava/lang/Object;Ljava/lang/Object;)V+16
j
java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V+56
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShuffleInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;)V+18
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.closeAndDeletePartitionsIfNeeded(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+25
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$applicationRemoved$5(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+3
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$319.run()V+12
j
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.dylib+0x289d70]
V [libjvm.dylib+0x2887fc]
V [libjvm.dylib+0x288a50]
V [libjvm.dylib+0x3118d0]
V [libjvm.dylib+0x524408]
V [libjvm.dylib+0x5242a4]
V [libjvm.dylib+0x456874]
C [libsystem_pthread.dylib+0x7878] _pthread_start+0x140
C [libsystem_pthread.dylib+0x25e0] thread_start+0x8
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.delete(J[BII)V+0
j org.rocksdb.RocksDB.delete([B)V+9
j org.apache.spark.network.shuffledb.RocksDB.delete([B)V+5
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShufflePartitionInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppAttemptShuffleMergeId;)V+16
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$removeAppShuffleInfoFromDB$8(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Ljava/lang/Integer;Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleMergePartitionsInfo;)V+24
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$321.accept(Ljava/lang/Object;Ljava/lang/Object;)V+16
j
java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V+56
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShuffleInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;)V+18
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.closeAndDeletePartitionsIfNeeded(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+25
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$applicationRemoved$5(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+3
j
org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$319.run()V+12
j
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
```
[hs_err_pid12358.log](https://github.com/apache/spark/files/9392790/hs_err_pid12358.log)
We may have operated a closed DB due to `close` and
'removeappshufflepartitioninfofromdb' are on different threads,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
