wangbing505 commented on issue #12456:
URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-5907224237
mat Path to GC Roots
first
```text
Class Name
| Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @
0x86295230
| 88 | 276,432
'- classLoader org.apache.hadoop.conf.Configuration @ 0x851a6070
| 40 | 240
'- conf org.apache.hadoop.io.Text$Comparator @ 0x851a6050
| 32 | 32
'- val java.util.concurrent.ConcurrentHashMap$Node @ 0x851a6030
| 32 | 64
'- [0] java.util.concurrent.ConcurrentHashMap$Node[16] @ 0x851a5fe0
| 80 | 144
'- table java.util.concurrent.ConcurrentHashMap @ 0x851a5fa0
| 64 | 208
'- comparators class org.apache.hadoop.io.WritableComparator
@ 0x862886f0
| 8 | 240
'- [1656] java.lang.Object[10240] @ 0x85d34e98
| 40,976 | 2,673,208
'- elementData java.util.Vector @ 0x83ca5d30
| 32 | 2,673,240
'- classes sun.misc.Launcher$AppClassLoader @
0x83d1ff58
| 88 | 3,575,792
|- scl class java.lang.ClassLoader @ 0x83d10380
System Class
| 40 | 2,152
|- scloader class sun.launcher.LauncherHelper @
0x83d15790 System Class
| 80 | 816
|- contextClassLoader
org.apache.logging.log4j.core.util.Log4jThread @ 0x83d65268
Log4j2-TF-4-Scheduled-2 Thread |
120 | 512
|- contextClassLoader
org.apache.logging.log4j.core.util.Log4jThread @ 0x83d654a8
Log4j2-TF-1-AsyncLogger[AsyncContext@2626b418]-1 Thread |
120 | 30,232
|- contextClassLoader java.lang.Thread @
0x83d65c40 Signal Dispatcher Thread
| 120 | 256
|- contextClassLoader java.lang.Thread @
0x86300fb0 arthas-command-execute Thread
| 120 | 864
|- contextClassLoader
com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalThread @
0x863011f0 arthas-NettyHttpTelnetBootstrap-3-2 Thread | 128 |
9,184
|- contextClassLoader java.lang.Thread @
0x86301510 mcp-keep-alive-scheduler Thread
| 120 | 200
|- contextClassLoader java.lang.Thread @
0x86301750 arthas-session-manager Thread
| 120 | 200
|- contextClassLoader java.lang.Thread @
0x86301990 arthas-shell-server Thread
| 120 | 200
|- contextClassLoader
com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalThread @
0x86301bd0 arthas-NettyWebsocketTtyBootstrap-4-2 Thread| 128 |
2,192
|- contextClassLoader
com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalThread @
0x86301ef0 arthas-NettyWebsocketTtyBootstrap-4-1 Thread| 128 |
2,192
|- contextClassLoader
com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalThread @
0x86302210 arthas-NettyHttpTelnetBootstrap-3-1 Thread | 128 |
2,464
|- contextClassLoader java.util.TimerThread @
0x86302530 arthas-timer Thread
| 128 | 208
|- contextClassLoader java.lang.Thread @
0x863026c8 Attach Listener Thread
| 120 | 680
|- contextClassLoader
com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread @
0x863028e0 hz.main.cached.thread-11 Thread | 136 | 4,896
|- contextClassLoader
com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread @
0x86302a98 hz.main.cached.thread-6 Thread | 136 | 4,896
|- contextClassLoader
com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread @
0x86302c50 hz.main.cached.thread-16 Thread | 136 | 4,896
|- contextClassLoader
org.apache.logging.log4j.core.util.Log4jThread @ 0x86302e08
Log4j2-TF-4-Scheduled-3 Thread |
120 | 512
|- contextClassLoader
com.hazelcast.internal.diagnostics.HealthMonitor$HealthMonitorThread @
0x86303128 hz.main.HealthMonitor Thread | 136 |
440
|- contextClassLoader java.lang.Thread @
0x863032b0 Connector-Scheduler-10c47c79-1 Thread
| 120 | 376
|- contextClassLoader java.lang.Thread @
0x863034f0 hz.main.TcpServer.thread-4 Thread
| 120 | 432
|- contextClassLoader java.lang.Thread @
0x86303750 hz.main.TcpServer.thread-3 Thread
| 120 | 432
|- contextClassLoader java.lang.Thread @
0x86303918 hz.main.TcpServer.thread-2 Thread
| 120 | 432
|- contextClassLoader java.lang.Thread @
0x86303ae0 hz.main.TcpServer.thread-1 Thread
| 120 | 432
|- contextClassLoader
com.hazelcast.internal.server.tcp.TcpServerAcceptor$AcceptorIOThread @
0x86303ca8 hz.main.IO.thread-Acceptor Thread | 128 |
384
|- contextClassLoader
com.hazelcast.internal.networking.nio.iobalancer.IOBalancerThread @ 0x86303f00
hz.main.IO.BalancerThread Thread | 144 | 456
|- contextClassLoader
com.hazelcast.internal.networking.nio.NioThread @ 0x863040f8
hz.main.IO.thread-out-2 Thread |
216 | 4,872
|- contextClassLoader
com.hazelcast.internal.networking.nio.NioThread @ 0x86304388
hz.main.IO.thread-out-1 Thread |
216 | 4,872
|- contextClassLoader
com.hazelcast.internal.networking.nio.NioThread @ 0x86304618
hz.main.IO.thread-out-0 Thread |
216 | 4,848
'- Total: 30 entries
| |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
second
```text
Class Name
| Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @
0x8629fc70 | 88 | 165,488
|- <classloader> class com.mongodb.internal.thread.DaemonThreadFactory @
0x862ae620 | 8 | 48
| '- <class> com.mongodb.internal.thread.DaemonThreadFactory @ 0x855c06f0
| 24 | 136
| '- threadFactory java.util.concurrent.ScheduledThreadPoolExecutor @
0x86300e18 | 80 | 480
| |- <Java Local> java.lang.Thread @ 0x86300c68
BufferPoolPruner-1-thread-1 Thread | 120 |
640
| |- this$0 java.util.concurrent.ThreadPoolExecutor$Worker @
0x86300e78 | 48 | 48
| |- this$0
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask @
0x855c07c8 | 72 | 112
| |- referent java.lang.ref.Finalizer @ 0x86158a18
| 40 | 40
| '- Total: 4 entries
| |
|- classloader java.security.ProtectionDomain @ 0x8630f530
| 40 | 400
| '- [1] java.security.ProtectionDomain[2] @ 0x855c08c0
| 24 | 24
| '- context java.security.AccessControlContext @ 0x855c0898
| 40 | 64
| '- inheritedAccessControlContext java.lang.Thread @ 0x86300c68
BufferPoolPruner-1-thread-1 Thread| 120 | 640
'- Total: 2 entries
| |
------------------------------------------------------------------------------------------------------------------------------------------
```
Collected using heapdump --live (i.e., the set of objects that remain alive
after GC).
The heap dump was taken after the job had completely finished. Both of the
following ClassLoader instances were present in the live object set, meaning
they remained alive even after a full GC had occurred after the job finished.
Leak #1 — Retained by a static field (Hadoop WritableComparator)
MAT “Path to GC Roots”, with weak/soft/phantom references excluded.
The following path is expanded from the GC Root toward the retained object,
with the strong-reference field name annotated at each hop:
```text
[GC Root] sun.misc.Launcher$AppClassLoader @ 0x83d1ff58
--(field classes)--> java.util.Vector @ 0x83ca5d30
--(field elementData)--> java.lang.Object[10240] @ 0x85d34e98
[index 1656]
--> class
org.apache.hadoop.io.WritableComparator @ 0x862886f0
--(static field comparators)--> java.util.concurrent.ConcurrentHashMap @
0x851a5fa0 <== static holder
--(field table)--> ConcurrentHashMap$Node[16] @ 0x851a5fe0
[index 0]
--(field val)--> ConcurrentHashMap$Node @ 0x851a6030
--> org.apache.hadoop.io.Text$Comparator @
0x851a6050
--(field conf)--> org.apache.hadoop.conf.Configuration @
0x851a6070
--(field classLoader)--> SeaTunnelChildFirstClassLoader @ 0x86295230
<== leaked object
(shallow 88 / retained 276,432)
Retention type: Static cache
Static holder: org.apache.hadoop.io.WritableComparator.comparators (static
ConcurrentHashMap)
```
Leak #2 — Retained by a live thread (MongoDB Driver BufferPoolPruner)
The second surviving ClassLoader instance in the same dump.
MAT “Path to GC Roots”, with weak/soft/phantom references excluded.
There are two independent strong-reference paths, but both ultimately
originate from the same live thread:
```text
[GC Root] java.lang.Thread @ 0x86300c68 "BufferPoolPruner-1-thread-1"
(alive)
|
|-- Branch A --
|
--(Java Local)-->
java.util.concurrent.ScheduledThreadPoolExecutor @ 0x86300e18
--(field threadFactory)-->
com.mongodb.internal.thread.DaemonThreadFactory @ 0x855c06f0
--(<class>)--> class
com.mongodb.internal.thread.DaemonThreadFactory @ 0x862ae620
--(<classloader>)--> SeaTunnelChildFirstClassLoader @
0x8629fc70 <== leaked object
(shallow 88 / retained 165,488)
|
|-- Branch B --
|
--(field inheritedAccessControlContext)-->
java.security.AccessControlContext @ 0x855c0898
--(field context)--> java.security.ProtectionDomain[2] @
0x855c08c0 [index 1]
--> java.security.ProtectionDomain @
0x8630f530
--(field classloader)--> SeaTunnelChildFirstClassLoader @
0x8629fc70 <== same object
```
Retention type: Live thread
Thread: "BufferPoolPruner-1-thread-1" (still running after the job finished)
Static holder: None; this path does not involve a static field.
and, For another job execution, after performing another heap dump, the two
additional SeaTunnelChildFirstClassLoader instances have the same “Path to GC
Roots” in MAT. the final reference count is zero
@SEZ9 ClassLoader count and Metaspace usage before and after job execution.
```text
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
sun.misc.Launcher$AppClassLoader 1
10577
com.taobao.arthas.agent.ArthasClassloader 2
4764
BootstrapClassLoader 1
3703
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 2
2320
sun.reflect.DelegatingClassLoader
253 253
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 15 ms.
[arthas@9]$
[arthas@9]$ memory
Memory
used total max
usage
heap
937M 2048M
30688M 3.06%
g1_eden_space
856M 1244M -1
68.81%
g1_survivor_space
0K 0K -1
0.00%
g1_old_gen
81M 804M
30688M 0.27%
nonheap
156M 163M 3312M
4.73%
code_cache
26M 26M 240M
10.85%
metaspace
116M 122M 2048M
5.69%
compressed_class_space
13M 15M 1024M
1.37%
direct
546K 546K -
100.00%
mapped
4K 4K -
100.00%
[arthas@9]$
[arthas@9]$
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
sun.misc.Launcher$AppClassLoader 1
10606
com.taobao.arthas.agent.ArthasClassloader 2
4765
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 4
4638
BootstrapClassLoader 1
3704
sun.reflect.DelegatingClassLoader
279 279
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 27 ms.
[arthas@9]$
[arthas@9]$
[arthas@9]$ memory
Memory
used total max
usage
heap
1169M 2048M
30688M 3.81%
g1_eden_space
1088M 1244M -1
87.46%
g1_survivor_space
0K 0K -1
0.00%
g1_old_gen
81M 804M
30688M 0.27%
nonheap
172M 180M 3312M
5.21%
code_cache
27M 27M 240M
11.39%
metaspace
129M 136M 2048M
6.34%
compressed_class_space
15M 16M 1024M
1.51%
direct
640K 640K -
100.00%
mapped
4K 4K -
100.00%
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
