wangbing505 commented on issue #12456:
URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-5807180004
1. **Relationship with `classloader-cache-mode`**
It is currently difficult to provide the complete logs. However, based on
our analysis, the issue appears to be related to the `classloader-cache-mode`
configuration.
Previously, `classloader-cache-mode` was configured as `false`, mainly
because we wanted to support plugin updates without restarting the entire
cluster service.
We have now changed it to `true`, and so far the issue appears to be
resolved. We are continuing to monitor the cluster to confirm whether this
change can consistently prevent the problem.
2. **Java Version**
The current Java version is:
```text
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
```
3. **Arthas and MAT Analysis**
We used Arthas for further analysis. The most abnormal classloader is:
```text
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
```
The initial `classloader` output was:
```text
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
191 210597
sun.misc.Launcher$AppClassLoader 1
12042
BootstrapClassLoader 1
3712
com.taobao.arthas.agent.ArthasClassloader 1
2385
sun.reflect.DelegatingClassLoader
689 689
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 197 ms.
```
We then executed:
```text
vmtool --action forceGc
```
However, there was no obvious change in the number of
`SeaTunnelChildFirstClassLoader` instances or the total number of loaded
classes.
After the forced GC, the output became:
```text
[arthas@9]$ classloader
name numberOfInstances loadedCountTotal
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 225
250131
sun.misc.Launcher$AppClassLoader 1 12046
BootstrapClassLoader 1 3757
com.taobao.arthas.agent.ArthasClassloader 1 3473
sun.reflect.DelegatingClassLoader 894 894
sun.misc.Launcher$ExtClassLoader 1 44
sun.reflect.misc.MethodUtil 1 1
Affect(row-cnt:7)
cost in 157 ms.
```
This suggests that the `SeaTunnelChildFirstClassLoader` instances and the
classes loaded by them were not reclaimed by GC. The number of classloader
instances and loaded classes actually continued to increase.
We also used MAT to analyze the heap dump and investigate the references
retaining these classloaders. a litter demo
The MAT analysis results are shown in the attached screenshots:
<img width="1208" height="284" alt="Image"
src="https://github.com/user-attachments/assets/cfd31507-5b62-40b6-83c3-f0166abfc27b";
/>
<img width="1038" height="197" alt="Image"
src="https://github.com/user-attachments/assets/110dd963-c9e1-4c0b-a949-7d7afa642d18";
/>
<img width="1066" height="181" alt="Image"
src="https://github.com/user-attachments/assets/76f56616-2929-4142-be8e-3655ed2a793f";
/>
<img width="1161" height="163" alt="Image"
src="https://github.com/user-attachments/assets/c01997b2-13bd-45a4-9bf9-5e94b9e7665a";
/>
4. **Currently Installed Connectors**
The currently installed connectors are:
```text
connectors["connector-cdc-base"]="seatunnel-connectors-v2/connector-cdc/connector-cdc-base"
connectors["connector-cdc-mysql"]="seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql"
connectors["connector-cdc-mongodb"]="seatunnel-connectors-v2/connector-cdc/connector-cdc-mongodb"
# connectors["connector-iceberg"]="seatunnel-connectors-v2/connector-iceberg"
# connectors["connector-hudi"]="seatunnel-connectors-v2/connector-hudi"
# connectors["connector-paimon"]="seatunnel-connectors-v2/connector-paimon"
# connectors["connector-fake"]="seatunnel-connectors-v2/connector-fake"
# connectors["connector-jdbc"]="seatunnel-connectors-v2/connector-jdbc"
connectors["connector-hive"]="seatunnel-connectors-v2/connector-hive"
# connectors["connector-console"]="seatunnel-connectors-v2/connector-console"
# connectors["connector-kafka"]="seatunnel-connectors-v2/connector-kafka"
connectors["connector-file-hadoop"]="seatunnel-connectors-v2/connector-file/connector-file-hadoop"
connectors["connector-file-ftp"]="seatunnel-connectors-v2/connector-file/connector-file-ftp"
connectors["connector-file-obs"]="seatunnel-connectors-v2/connector-file/connector-file-obs"
connectors["connector-file-sftp"]="seatunnel-connectors-v2/connector-file/connector-file-sftp"
# connectors["connector-mongodb"]="seatunnel-connectors-v2/connector-mongodb"
#
connectors["connector-elasticsearch"]="seatunnel-connectors-v2/connector-elasticsearch"
```
At the moment, the only type of job we are submitting is:
```text
MongoDB -> Hive
```
Based on the current observations, we suspect that when
`classloader-cache-mode=false`, plugin classloaders may be continuously created
during job submission/execution and are not being garbage-collected as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
