dingshun3016 commented on PR #2504:
URL: https://github.com/apache/paimon/pull/2504#issuecomment-3345661215
Found one Java-level deadlock:
=============================
"MANIFEST-READ-THREAD-POOL-t1":
waiting to lock monitor 0x000014bfd4a08c98 (object 0x00000003c09c8b28, a
java.lang.Object),
which is held by UNKNOWN_owner_addr=0x000014bfd6782000
Java stack information for the threads listed above:
===================================================
"MANIFEST-READ-THREAD-POOL-t1":
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1607)
- waiting to lock <0x00000003c09c8b28> (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:320)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:426)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:169)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:161)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:99)
- locked <0x00000003ea560540> (a
org.apache.hadoop.io.retry.RetryInvocationHandler$Call)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:363)
at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:933)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:922)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:911)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1078)
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:331)
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:327)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:902)
at
org.apache.paimon.fs.hadoop.HadoopFileIO.newInputStream(HadoopFileIO.java:76)
at
org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:82)
at
org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.<init>(AvroBulkFormat.java:70)
at
org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.<init>(AvroBulkFormat.java:57)
at
org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:54)
at
org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:130)
at
org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:180)
at
org.apache.paimon.utils.ObjectsFile$$Lambda$387/161605449.apply(Unknown Source)
at org.apache.paimon.utils.ObjectsCache.read(ObjectsCache.java:98)
at
org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:149)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:134)
at
org.apache.paimon.operation.AbstractFileStoreScan.readManifest(AbstractFileStoreScan.java:433)
at
org.apache.paimon.operation.AbstractFileStoreScan.lambda$readAndMergeFileEntries$10(AbstractFileStoreScan.java:368)
at
org.apache.paimon.operation.AbstractFileStoreScan$$Lambda$413/158556042.apply(Unknown
Source)
at
org.apache.paimon.utils.ThreadPoolUtils.lambda$randomlyExecuteSequentialReturn$2(ThreadPoolUtils.java:163)
at
org.apache.paimon.utils.ThreadPoolUtils$$Lambda$414/649967112.call(Unknown
Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
A deadlock occurred in the production environment. I wonder if it is related
to the cache FileSystem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
