DanielLeens commented on issue #12456:
URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-6073517148
Classification: A / Zeta class-loader lifecycle and Metaspace.
@wangbing505 thank you for retesting #12626 and for the new root paths. To
keep the scope honest: #12626 is a partial change. It only releases the MongoDB
`PowerOfTwoBufferPool` pruner and the Hadoop `WritableComparator.comparators`
configuration. It is not a general class-loader-leak fix, and I am not adding
more cleanup to it until the data below shows which remaining roots actually
grow per job.
What the library sources say (Hadoop 3.1.4, MongoDB driver 4.7.1; this is
source reading, not a reproduction):
1. `Token.renewers` is a static `ServiceLoader` created in `Token`'s static
initializer. It captures the thread context class loader once, when `Token` is
first initialized. It can pin one job loader for the life of the JVM, but it
does not grow per job.
2. `Client.ClientExecutorServiceFactory` creates a static cached thread pool
("IPC Parameter Sending Thread #N", 60s idle timeout). A pooled thread keeps
the access control context of the thread that created it, so it pins the loader
of the job that created it while it lives. That is also one loader at a time,
not one per job.
3. The `DaemonThreadFactory` path in your second dump goes only through
`java.lang.ref.Finalizer` / JNA `CleanerRef` `discovered` links. A
`ScheduledThreadPoolExecutor` is finalizable on Java 8, so this looks like an
executor that has been shut down (the pruner is stopped by #12626) and is
waiting for finalization, not a live root. This is an inference; Arthas
`forceGc` runs a single GC, and finalizable objects need a further cycle.
None of this shows that the `SeaTunnelChildFirstClassLoader` count still
grows per job on the #12626 build, which is what drives the Metaspace OOM.
Please provide, from one worker running the #12626 build with cache mode false:
1. The `SeaTunnelChildFirstClassLoader` count and Metaspace used after 0, 5
and 10 sequential identical MongoDB-to-Hive jobs, each after a forced GC.
2. A heap dump taken with `jmap -dump:live` after two GCs and
`System.runFinalization()` (for example via `vmtool`), and the path to GC roots
for one loader that is still present.
3. A thread dump after the last job showing whether the IPC Parameter
Sending Thread is alive and its context class loader.
If the count is flat or grows by a small constant, the remaining roots are
bounded and the remaining work is documentation or a targeted single-owner fix.
If it still grows per job, the retaining path from the dump will tell us which
owner to close next, with a real-library regression.
`classloader-cache-mode=true` stays a valid mitigation for identical JAR sets.
No label or assignment change in this pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
