tracy1014-hub commented on issue #65416: URL: https://github.com/apache/doris/issues/65416#issuecomment-4933733446
Hi @yiguolei — `jstack` and `jmap` captured. You're spot-on: these are Java threads in the embedded JVM inside `doris_be`. Public gist: **https://gist.github.com/tracy1014-hub/1af1032ea8b8cc08b6cf3d28d790abfe** Files: - `jstack-doris-be-0.txt.gz.b64` — full `jstack 994` output (15,080 Java threads). Decode: `base64 -d file.b64 | gunzip > jstack.txt`. jstack ran in 6 seconds total — vastly faster than gdb. - `jmap-histo-doris-be-0.txt` — `jmap -histo 994` output (Java object histogram). - `JVM-ANALYSIS.md` — analysis. ## Thread name distribution (15,080 total Java threads) ``` 7,324 sdk-ScheduledExecutor-N-M (47.7% of all Java threads) 51 Thread-N 1 each: main, Reference Handler, Finalizer, Signal Dispatcher, Service Thread, Monitor Deflation Thread, C2 CompilerThread0, Notification Thread, pool-1-thread-1, Sweeper thread, MutableQuantiles, mysql-cj-abandoned-connection-cleanup, org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner, Timer for 's3a-file-system' metrics system ``` 7,324 of the 15,080 Java threads are workers in `ScheduledThreadPoolExecutor` instances, all named `sdk-ScheduledExecutor-N-M` with unique (N, M) pairs. Each unique N is a distinct `ScheduledThreadPoolExecutor` instance — so we have **thousands of leaked executors** in the JVM. ## All 7,324 parked threads have an identical stack ``` Unsafe.park (Native Method) - parking to wait for <0x...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) LockSupport.park (LockSupport.java:341) AbstractQueuedSynchronizer$ConditionNode.block (AbstractQueuedSynchronizer.java:506) ForkJoinPool.unmanagedBlock (ForkJoinPool.java:3463) ForkJoinPool.managedBlock (ForkJoinPool.java:3434) AbstractQueuedSynchronizer$ConditionObject.await (AbstractQueuedSynchronizer.java:1623) ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:1177) ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:899) ThreadPoolExecutor.getTask (ThreadPoolExecutor.java:1062) ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1122) ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) java.lang.Thread.run (Thread.java:833) ``` All parked on `ScheduledThreadPoolExecutor$DelayedWorkQueue.take` — waiting for tasks that never arrive. The executors are never `shutdown()`'d, so their workers never exit. ## jmap histogram confirms the leak Top objects by bytes/instances: | # | instances | bytes | class | |---|---|---|---| | 1 | 569,928 | **521 MB** | `byte[]` | | 2 | **5,631,659** | 135 MB | **`org.apache.paimon.data.Timestamp`** | | 3 | 1,524,665 | 85 MB | `java.nio.HeapByteBuffer` | | 5 | 738,882 | 24 MB | `java.util.HashMap$Node` | | 6 | 634,381 | 20 MB | `javax.management.MBeanAttributeInfo` | | 8 | 634,314 | 15 MB | `javax.management.Attribute` | | 9 | 577,459 | 14 MB | `org.apache.hadoop.metrics2.impl.MetricCounterLong` | | 14 | 75,170 | 7.8 MB | `java.lang.management.ThreadInfo` | | 15 | **15,053** | 5.5 MB | **`java.lang.Thread`** | | 18 | 90,373 | 2.9 MB | `org.apache.paimon.shade.org.apache.parquet.io.api.Binary$ByteBufferBackedBinary` | | 20 | 61,335 | 2.0 MB | `org.apache.paimon.memory.MemorySegment` | Key signals: 1. **`java.lang.Thread` instances: 15,053** — matches the OS-level `rs_normal` thread count (15,174) within timing delta. **Byte-for-byte confirmation that the `rs_normal` OS threads are Java `Thread` objects**. 2. **`org.apache.paimon.data.Timestamp` 5.6 M instances** — Paimon allocates a Timestamp per row (or per scan?) and they're not being GC'd. Suggests a `ThreadLocal` cache or per-scan `HashMap` that holds Timestamps and is never cleared. 3. **`javax.management.MBeanAttributeInfo` 634k + `MetricCounterLong` 577k** — Hadoop metrics2 MBeans. The S3A client registers MBeans per `S3AFileSystem` instance. **634k MBean attributes is consistent with thousands of distinct S3A client instances** — every Paimon scan creates a new S3A client → new MBean registration → never unregistered. 4. **`byte[]` 521 MB across 570k instances** — Paimon data buffers / Parquet column payloads. 5. **`AQS$ConditionNode` ~25k instances** (in the full histogram) — corresponds to the ~15k parked workers (each holds one AQS$ConditionNode). ## Diagnosis The leak path: 1. Every external scan against the Paimon catalog creates a fresh Hadoop `S3AFileSystem` instance (with its own S3 client, `ScheduledThreadPoolExecutor` for connection pool management, MBean registration for Hadoop metrics2, etc.). 2. The `S3AFileSystem` is never `close()`'d, so its `ScheduledThreadPoolExecutor` is never `shutdown()`'d, and its MBeans are never unregistered. 3. Each leaked executor creates at least one worker thread named `sdk-ScheduledExecutor-N-M`, parked on `DelayedWorkQueue.take` waiting for tasks that never come. 4. Over hours of sustained Paimon scan traffic, the count of leaked `ScheduledThreadPoolExecutor` instances grows linearly with query rate. By 4-5 hours, we have 7,324 leaked executors → 15k parked worker threads → ~5-7 GB RSS just for thread stacks → freeze or OOM. The `jni.log` from earlier also showed this pattern — every Paimon scan re-runs `HadoopSecuredFileSystem.trySecureFileSystem` + `HadoopModule.install` + `CodecPool.getDecompressor` from scratch, no caching of the FileSystem or compressor across scans. ## Where to look in the code The fix has to be on the **Paimon JNI scanner side**, not in the C++ remote scanner pool. Likely places to look: 1. Paimon scanner JARs under `$DORIS_HOME/lib/java_extensions/paimon-scanner/` — search for classes that create `S3AFileSystem`, `AmazonS3Client`, `S3Client`, or call `FileSystem.get(...)`. Each call site should be checked for whether the returned FileSystem is cached/shared vs. created fresh. 2. The Hadoop `FileSystem` class has a global `FileSystem.CACHE` keyed by URI + UserGroupInformation — if every Paimon scan uses a slightly different URI (e.g., different query parameters, different path prefix), the cache will hold a new entry per scan, and each entry holds a `ScheduledThreadPoolExecutor`. Worth checking `org.apache.hadoop.fs.FileSystem$Cache$CacheCleaner` activity and the size of `FileSystem.CACHE`. 3. Paimon-side fix: reuse a shared `S3AFileSystem` (or at least call `close()` / `shutdown()` on the per-scan instance when the scan completes). The C++ remote scanner pool is healthy and bounded (`max_threads{rs_normal}=256`, `active=0`, `task_execution_count=4,798`). The leak is entirely on the JVM side. ## WeChat Will DM you about this separately — I want to keep the technical thread clean. ## Live state `doris-be-0` (the experiment BE with `num_cores = 8`) is still running and growing. Latest monitor sample: rs_normal=15,174, threads=17,197 at T+3h38m. It's on track to hit the natural ~27k-thread freeze in roughly 2-3 more hours. I'm happy to either let it run to capture the natural-freeze `be.INFO`, or rotate it now that we've nailed the diagnosis — your call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
