Doris-Breakwater commented on issue #67898: URL: https://github.com/apache/doris/issues/67898#issuecomment-5645312334
Initial triage: the evidence is internally consistent and this should be treated as a high-severity availability/resource-exhaustion problem. It is also a near-exact duplicate of #65416, which reports the same unbounded `rs_normal` growth for Paimon-on-S3 on 4.1.1-rc01, including ineffective BE and workload-group caps. This report adds a much better controlled reproducer, so the two issues should be linked and investigated together. ### What is verified from the reported revision I checked the exact build commit, `7126cf65d96ebc43fce0906f51e92c1a2ccf24a6`: - `rs_normal` is the remote scan scheduler owned by the `normal` workload group ([construction path](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/runtime/workload_group/workload_group.cpp#L582-L595)). Having only one workload group narrows the problem, but does not bypass this workload-group-owned scheduler. - The raw default of `doris_max_remote_scanner_thread_pool_thread_num` is indeed `-1` ([config](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/common/config.cpp#L325-L333)), but `-1` is an automatic-sizing sentinel, not an unlimited value: it becomes `max(512, detected_cores * 10)` ([calculation](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/exec/scan/scanner_scheduler.cpp#L381-L390)). Thus `/api/show_config = -1` and the documented effective floor of 512 are compatible, although the documentation should explain the sentinel. - With the shown local size of 48, an explicit BE value of 256 should yield a 256 maximum unless the `normal` workload group supplies a positive `max_remote_scan_thread_num`, which overrides the BE fallback ([override path](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/runtime/workload_group/workload_group.cpp#L397-L408)). - The task executor only creates a worker when `num_threads + pending_start < max_threads` ([cap check](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L413-L437)), and idle workers above the minimum are supposed to retire after 500 ms ([retirement path](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L523-L548)). A single correctly accounted pool therefore cannot intentionally contain 3,192 or 35,000 live workers. This confirms that the observed behavior violates the intended cap/lifecycle. It does **not** yet identify which invariant is failing. The remaining possibilities include scheduler accounting/teardown, multiple same-named scheduler owners, or non-Doris/JVM-side threads retaining or inheriting the native name. The thread-name histogram alone cannot distinguish them, so I would not assign a Paimon or scheduler root cause yet. ### Highest-value missing evidence Please collect the following on a freshly restarted BE, immediately before and after the 20-query reproducer: 1. `SHOW WORKLOAD GROUPS` output for `normal`, especially `max_remote_scan_thread_num` and `min_remote_scan_thread_num`. 2. The BE INFO lines containing `[topic_publish_wg] ... thread info=`. This revision already logs remote-pool accounting as `r_scan num:<accounted>, real_num:<thread-set-size>, min_num:<...>, max_num:<...>` ([format](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/runtime/workload_group/workload_group.cpp#L679-L700)). The before/after values are the most direct test of whether the executor itself sees the leaked threads. 3. `/threadz` and `/threadz?group=thread%20pool` before/after. Comparing Doris-registered thread counts with `/proc/<pid>/task` will show whether all `rs_normal` TIDs were created by Doris's thread wrapper. 4. For 5 newly created and 5 old `rs_normal` TIDs: `/proc/<pid>/task/<tid>/status`, `wchan`, and `stack` (if permitted), plus native backtraces. Also provide `jcmd <pid> Thread.print -l` or `jstack <pid>` and map those decimal TIDs to Java `nid=0x...`. This will establish whether they are idle in the scheduler, blocked in JNI/Paimon/Hadoop/S3 code, or absent from the JVM. 5. The Paimon table type (append-only vs primary-key), file format (ORC/Parquet), bucket mode, approximate split count for the test query, and whether every query completed normally or any were cancelled/timed out. ### Suggested next steps - Link/consolidate with #65416 and route jointly to BE execution/workload scheduling and Lakehouse/Paimon maintainers. No labels are currently set; `kind/bug` and `area/catalog` are appropriate, plus the repository's high-severity/availability label if applicable. - Use `r_scan num/real_num/max_num` plus representative stacks to choose the owner before changing code. If `real_num` stays at or below 256 while `/proc` grows, investigate JNI/plugin-created threads or native-name inheritance. If `real_num` itself exceeds `max_num`, the task-executor accounting invariant is directly broken. If several pool instances appear, trace workload-group scheduler creation/destruction. - Run one staging A/B on the same table with `SET enable_paimon_cpp_reader=true` if that reader supports this table. Growth only on the default JNI path would isolate Paimon/JNI lifecycle from the common remote scheduler; growth on both paths would point back toward scheduler ownership/accounting. - The reported build predates Paimon observability and cleanup work in #65354, #65365, and #65503. A repeat of the exact 20-query test on 4.1.4-rc04 is useful because that source contains the relevant async-reader metrics/control and iterator/JNI cleanup changes. This is a diagnostic upgrade test, **not** a claim that the issue is already fixed. - Until ownership is confirmed, rolling BE restarts before thread/RSS exhaustion are the only demonstrated mitigation here. The 256 cap and allocator tuning should not be presented as fixes for the observed behavior. Breakwater-GitHub-Analysis-Slot: slot_52e24c6dcbe1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
