wangbing505 commented on issue #12456: URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-6120517443
ClassLoader counts and memory usage after 0 jobs <img width="1398" height="416" alt="Image" src="https://github.com/user-attachments/assets/9aeb62dd-fb02-4375-a13e-9934c268ab78"; /> ClassLoader counts and memory usage after 5 jobs <img width="1445" height="474" alt="Image" src="https://github.com/user-attachments/assets/a0b418a5-b388-4699-9418-241c810a32f4"; /> ClassLoader counts and memory usage after 10 jobs <img width="1471" height="406" alt="Image" src="https://github.com/user-attachments/assets/4e549d79-8347-46a7-af28-6fdbb47dae8d"; /> 2、Path to GC Roots: I checked the Path to GC Roots for each of these 20 SeaTunnelChildFirstClassLoader instances individually. The following two paths were duplicated; ```text Class Name | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @ 0x865f1c40 | 88 | 206,848 '- <classloader> class org.apache.seatunnel.shade.connector.hive.org.apache.parquet.hadoop.codec.SnappyCodec @ 0x8661ad40 | 8 | 32 '- key java.util.concurrent.ConcurrentHashMap$Node @ 0x864a9df0 | 32 | 248 '- next java.util.concurrent.ConcurrentHashMap$Node @ 0x864a9d00 | 32 | 1,056 '- [12] java.util.concurrent.ConcurrentHashMap$Node[32] @ 0x86e3bd88 | 144 | 6,352 '- table java.util.concurrent.ConcurrentHashMap @ 0x8529b0b8 | 64 | 6,416 '- CONSTRUCTOR_CACHE class org.apache.hadoop.util.ReflectionUtils @ 0x850d3038 | 32 | 6,584 '- [1542] java.lang.Object[10240] @ 0x86433b40 | 40,976 | 2,995,048 '- elementData java.util.Vector @ 0x83ca45d8 | 32 | 2,995,080 '- classes sun.misc.Launcher$AppClassLoader @ 0x84117bf0 | 88 | 3,898,040 |- contextClassLoader com.hazelcast.internal.diagnostics.HealthMonitor$HealthMonitorThread @ 0x83d57068 hz.main.HealthMonitor Thread | 136 | 440 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector$DetectorThread @ 0x83d572b0 hz.main.SlowOperationDetectorThread Thread| 136 | 536 |- contextClassLoader com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier$ResponseThread @ 0x83d57438 hz.main.response-1 Thread | 144 | 2,576 |- contextClassLoader com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier$ResponseThread @ 0x83d57600 hz.main.response-0 Thread | 144 | 2,576 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d577c8 hz.main.generic-operation.thread-49 Thread | 184 | 5,016 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d57a00 hz.main.generic-operation.thread-48 Thread | 184 | 6,328 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d57bb8 hz.main.generic-operation.thread-47 Thread | 184 | 6,248 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d57d70 hz.main.generic-operation.thread-46 Thread | 184 | 49,608 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d57f28 hz.main.generic-operation.thread-45 Thread | 184 | 6,432 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d580e0 hz.main.generic-operation.thread-44 Thread | 184 | 5,496 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58298 hz.main.generic-operation.thread-43 Thread | 184 | 6,432 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58450 hz.main.generic-operation.thread-42 Thread | 184 | 5,208 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58608 hz.main.generic-operation.thread-41 Thread | 184 | 5,200 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d587c0 hz.main.generic-operation.thread-40 Thread | 184 | 5,312 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58978 hz.main.generic-operation.thread-39 Thread | 184 | 6,456 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58b30 hz.main.generic-operation.thread-38 Thread | 184 | 50,000 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58ce8 hz.main.generic-operation.thread-37 Thread | 184 | 49,504 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58ea0 hz.main.generic-operation.thread-36 Thread | 184 | 49,504 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59058 hz.main.generic-operation.thread-35 Thread | 184 | 5,304 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59210 hz.main.generic-operation.thread-34 Thread | 184 | 6,000 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d593c8 hz.main.generic-operation.thread-33 Thread | 184 | 5,240 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59580 hz.main.generic-operation.thread-32 Thread | 184 | 49,752 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59738 hz.main.generic-operation.thread-31 Thread | 184 | 6,616 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d598f0 hz.main.generic-operation.thread-30 Thread | 184 | 6,720 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59aa8 hz.main.generic-operation.thread-29 Thread | 184 | 5,496 '- Total: 25 of 30 entries; 5 more | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ``` ```text Class Name | Shallow Heap | Retained Heap --------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @ 0x865d1be8 | 88 | 120,976 '- <classloader> class com.mongodb.internal.thread.DaemonThreadFactory @ 0x865ea110 | 8 | 48 '- <class> com.mongodb.internal.thread.DaemonThreadFactory @ 0x85dd8c20 | 24 | 136 '- threadFactory java.util.concurrent.ScheduledThreadPoolExecutor @ 0x868660c8 | 80 | 680 '- referent java.lang.ref.Finalizer @ 0x86263d70 | 40 | 9,160 '- next java.lang.ref.Finalizer @ 0x869bf968 | 40 | 9,200 '- next java.lang.ref.Finalizer @ 0x86e3c9e8 | 40 | 9,240 '- next java.lang.ref.Finalizer @ 0x86e3ca10 | 40 | 9,280 '- next java.lang.ref.Finalizer @ 0x86e3ca38 | 40 | 9,320 '- next java.lang.ref.Finalizer @ 0x86e3ca60 | 40 | 9,360 '- next java.lang.ref.Finalizer @ 0x86e3ca88 | 40 | 9,400 '- next java.lang.ref.Finalizer @ 0x86e3cab0 | 40 | 9,440 '- next java.lang.ref.Finalizer @ 0x86e3cad8 | 40 | 9,480 '- next java.lang.ref.Finalizer @ 0x86e3cb00 | 40 | 9,520 '- next java.lang.ref.Finalizer @ 0x86e3cb28 | 40 | 9,560 '- next java.lang.ref.Finalizer @ 0x86e3cb50 | 40 | 9,600 '- next java.lang.ref.Finalizer @ 0x86e3cb78 | 40 | 9,640 '- next java.lang.ref.Finalizer @ 0x86e3cba0 | 40 | 9,680 '- next java.lang.ref.Finalizer @ 0x86e3cbc8 | 40 | 9,720 '- next java.lang.ref.Finalizer @ 0x86e3cbf0 | 40 | 9,760 '- next java.lang.ref.Finalizer @ 0x86e3cc18 | 40 | 9,800 '- next java.lang.ref.Finalizer @ 0x86e3cc40 | 40 | 9,840 '- next java.lang.ref.Finalizer @ 0x86e3cc68 | 40 | 9,880 '- next java.lang.ref.Finalizer @ 0x86e3cc90 | 40 | 9,920 '- next java.lang.ref.Finalizer @ 0x86e3ccb8 | 40 | 9,960 '- next java.lang.ref.Finalizer @ 0x86e3cce0| 40 | 10,000 --------------------------------------------------------------------------------------------------------------------------------------------------- ``` each of the remaining paths appeared only once. ```text Class Name | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @ 0x866ce9c0 | 88 | 207,104 '- loader java.util.ServiceLoader @ 0x858c3f00 | 32 | 128 '- renewers class org.apache.hadoop.security.token.Token @ 0x867f2eb0 | 24 | 216 '- [5937] java.lang.Object[10240] @ 0x86433b40 | 40,976 | 2,995,048 '- elementData java.util.Vector @ 0x83ca45d8 | 32 | 2,995,080 '- classes sun.misc.Launcher$AppClassLoader @ 0x84117bf0 | 88 | 3,898,040 |- contextClassLoader com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier$ResponseThread @ 0x83d57600 hz.main.response-0 Thread | 144 | 2,576 |- contextClassLoader com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier$ResponseThread @ 0x83d57438 hz.main.response-1 Thread | 144 | 2,576 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector$DetectorThread @ 0x83d572b0 hz.main.SlowOperationDetectorThread Thread| 136 | 536 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d5a340 hz.main.generic-operation.thread-24 Thread | 184 | 6,328 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d5a188 hz.main.generic-operation.thread-25 Thread | 184 | 6,224 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59fd0 hz.main.generic-operation.thread-26 Thread | 184 | 6,168 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59e18 hz.main.generic-operation.thread-27 Thread | 184 | 49,816 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59c60 hz.main.generic-operation.thread-28 Thread | 184 | 6,328 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59aa8 hz.main.generic-operation.thread-29 Thread | 184 | 5,496 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d598f0 hz.main.generic-operation.thread-30 Thread | 184 | 6,720 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59738 hz.main.generic-operation.thread-31 Thread | 184 | 6,616 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59580 hz.main.generic-operation.thread-32 Thread | 184 | 49,752 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d593c8 hz.main.generic-operation.thread-33 Thread | 184 | 5,240 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59210 hz.main.generic-operation.thread-34 Thread | 184 | 6,000 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d59058 hz.main.generic-operation.thread-35 Thread | 184 | 5,304 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58ea0 hz.main.generic-operation.thread-36 Thread | 184 | 49,504 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58ce8 hz.main.generic-operation.thread-37 Thread | 184 | 49,504 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58b30 hz.main.generic-operation.thread-38 Thread | 184 | 50,000 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58978 hz.main.generic-operation.thread-39 Thread | 184 | 6,456 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d587c0 hz.main.generic-operation.thread-40 Thread | 184 | 5,312 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58608 hz.main.generic-operation.thread-41 Thread | 184 | 5,200 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58450 hz.main.generic-operation.thread-42 Thread | 184 | 5,208 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d58298 hz.main.generic-operation.thread-43 Thread | 184 | 6,432 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d580e0 hz.main.generic-operation.thread-44 Thread | 184 | 5,496 |- contextClassLoader com.hazelcast.spi.impl.operationexecutor.impl.GenericOperationThread @ 0x83d57f28 hz.main.generic-operation.thread-45 Thread | 184 | 6,432 '- Total: 25 of 30 entries; 5 more | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ``` ```text Class Name | Shallow Heap | Retained Heap ---------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @ 0x86cd3490 | 88 | 206,304 |- classloader java.security.ProtectionDomain @ 0x86e33e58 | 40 | 72 | '- [4] java.security.ProtectionDomain[5] @ 0x86f472c8 | 40 | 40 | '- context java.security.AccessControlContext @ 0x86f472a0 | 40 | 880 | '- inheritedAccessControlContext org.apache.hadoop.util.Daemon @ 0x86f470f8 LeaseRenewer:hueuser@ns-dev260 Thread| 128 | 1,528 |- <classloader> class org.apache.seatunnel.shade.connector.hive.org.apache.parquet.hadoop.codec.SnappyCodec @ 0x86cee4a0 | 8 | 32 | '- key java.util.concurrent.ConcurrentHashMap$Node @ 0x86e3c140 | 32 | 248 '- Total: 2 entries | | ---------------------------------------------------------------------------------------------------------------------------------------------------------- ``` ```text Class Name | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @ 0x8663b318 | 88 | 206,304 |- classloader java.security.ProtectionDomain @ 0x85dd7378 | 40 | 80 | '- [1] java.security.ProtectionDomain[5] @ 0x85dd7350 | 40 | 40 | '- context java.security.AccessControlContext @ 0x85dd7328 | 40 | 1,432 | '- inheritedAccessControlContext java.lang.Thread @ 0x86866118 IPC Parameter Sending Thread #1 Thread | 120 | 6,320 |- <classloader> class org.apache.seatunnel.shade.connector.hive.org.apache.parquet.hadoop.codec.SnappyCodec @ 0x86664178| 8 | 32 '- Total: 2 entries | | -------------------------------------------------------------------------------------------------------------------------------------------------------- ``` thread dump after the last job 执行命令 jstack -l <pid> > /tmp/threads-12626.txt grep -i "ipc\|Parameter Sending" /tmp/threads-12626.txt 没有找到ipc相关线程 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
