wangbing505 commented on issue #12456:
URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-5862488423
Thank you for your response.
### ClassLoader comparison before and after job submission
Before submitting the job:
```text
[arthas@9]$ classloader
name numberOfInstances
loadedCountTotal
sun.misc.Launcher$AppClassLoader 1 7657
BootstrapClassLoader 1 3454
com.taobao.arthas.agent.ArthasClassloader 1 2364
sun.reflect.DelegatingClassLoader 152 152
sun.misc.Launcher$ExtClassLoader 1 42
sun.reflect.misc.MethodUtil 1 1
Affect(row-cnt:6) cost in 11 ms.
```
After submitting the first job:
```text
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
sun.misc.Launcher$AppClassLoader 1
10586
BootstrapClassLoader 1
3693
com.taobao.arthas.agent.ArthasClassloader 1
2364
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 2
2320
sun.reflect.DelegatingClassLoader
245 245
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 13 ms.
```
After submitting another job:
```text
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
sun.misc.Launcher$AppClassLoader 1
10610
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 4
4643
BootstrapClassLoader 1
3694
com.taobao.arthas.agent.ArthasClassloader 1
2367
sun.reflect.DelegatingClassLoader
271 271
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 12 ms.
```
After another 10 minutes, with no new jobs submitted, the number of
`SeaTunnelChildFirstClassLoader` instances did not increase:
```text
[arthas@9]$ classloader
name
numberOfInstances loadedCountTotal
sun.misc.Launcher$AppClassLoader 1
10655
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 4
4645
BootstrapClassLoader 1
3694
com.taobao.arthas.agent.ArthasClassloader 1
2367
sun.reflect.DelegatingClassLoader
271 271
sun.misc.Launcher$ExtClassLoader 1
44
sun.reflect.misc.MethodUtil 1
1
Affect(row-cnt:7) cost in 18 ms.
```
This seems to indicate that the `SeaTunnelChildFirstClassLoader` instances
are primarily created when new jobs are submitted, rather than continuously
increasing during idle periods.
### Job information
The Job ID is:
```text
23184458863328
```
I used the following command to check the job status across the cluster:
```text
./bin/seatunnel.sh -j 23184458863328
```
The result was:
```json
{
"jobId": 23184458863328,
"jobName": "local_test_mongo2hive_0615_08_copy_copy",
"jobStatus": "FINISHED",
"submitTime": 1790558985229,
"startTime": 1790558986875,
"finishTime": 1790558992144,
"pipelineStateMapperMap": {
"PipelineLocation(jobId=23184458863328, pipelineId=1)": {
"pipelineStatus": "FINISHED",
"executionStateMap": {
"TaskGroupLocation{jobId=23184458863328, pipelineId=1,
taskGroupId=1}": "FINISHED",
"TaskGroupLocation{jobId=23184458863328, pipelineId=1,
taskGroupId=2}": "FINISHED",
"TaskGroupLocation{jobId=23184458863328, pipelineId=1,
taskGroupId=3}": "FINISHED"
}
}
},
"errorMessage": null,
"project": "dataex_dev_project"
}
```
log message
<img width="1634" height="210" alt="Image"
src="https://github.com/user-attachments/assets/8f13c251-c22b-4e17-805d-a3c6e28394b6";
/>
### Relevant logs and MAT analysis
I also took a heap dump at this point, then analyzed it using MAT.
first
<img width="1422" height="790" alt="Image"
src="https://github.com/user-attachments/assets/1bd0c366-b324-40cb-bec7-1ea572b84bb8";
/>
second
<img width="1325" height="411" alt="Image"
src="https://github.com/user-attachments/assets/6379748c-05d3-4eac-bcb0-fdcd2d05b919";
/>
The first retained `ChildFirstClassLoader` is particularly difficult to deal
with.
I previously tried to explicitly clean up these references inside
`releaseClassLoader`. However, after cleaning up one reference chain, another
reference from a different Hadoop class appeared and retained the ClassLoader
again.
The current analysis shows the following:
After removing A/B/C and checking the heap again, two new chains with
essentially the same pattern appeared:
* `CodecPool.compressorCounts`: this is still related to `CodecPool`, but
this time it is a Guava `LocalCache` rather than a `java.util.Map`. Therefore,
the previous `(Map)` cast and `removeClassKeyedEntriesLoadedBy` logic cannot
cover this case. This appears to be exactly why the same type of reference
shows up again in the latest MAT analysis.
* `WritableComparator.comparators`: another Hadoop static registry that
retains classes loaded by the child ClassLoader.
There seem to be many long-lived Hadoop static structures whose keys are
Classes loaded by the child ClassLoader, such as `SerializationFactory`,
`Groups`, and others.
The exact structures also vary depending on the Hadoop version. In some
versions they are regular Maps, while in others they may be Guava caches.
Handling each one individually would therefore require continuously adding
special cases.
After seeing the same pattern appear in multiple rounds of analysis, I think
this is becoming an architectural signal rather than an isolated reference leak.
Instead of continuing to remove individual references one by one, I am
considering a more fundamental approach: making the Parquet compression-related
classes be loaded by the parent ClassLoader, so that the keys in these
long-lived Hadoop static registries are associated with the parent ClassLoader
rather than the `SeaTunnelChildFirstClassLoader`. In that case, these static
registries would no longer retain the SeaTunnel child ClassLoader.
### Questions
Based on the above observations, do you have any recommended approaches for
addressing this ClassLoader retention problem?
In particular, I would like to understand whether we should:
1. Continue cleaning up individual Hadoop static references during
`releaseClassLoader`;
2. Change the ClassLoader relationship so that certain Hadoop/Parquet
classes are loaded by the parent ClassLoader;
3. Focus on identifying and stopping engine-side threads such as
`BufferPoolPruner` that retain the child ClassLoader;
4. Or consider a more systematic lifecycle/design change for
`SeaTunnelChildFirstClassLoader`.
Any suggestions on the recommended direction would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
