Doris-Breakwater commented on issue #70317: URL: https://github.com/apache/doris/issues/70317#issuecomment-6676246742
The reported client lifecycle defect is confirmed by source inspection. I checked the exact master commit `87b462e4f58104140f098f5f4a8b5d58bec70149` and the public `2.1.8-rc01` tag. Both borrow a peer FE client in `QueryStatsUtil.getStats()` without any corresponding return or invalidation. This supports treating the issue as an FE resource leak with potential cluster availability impact. I have not run a multi-FE reproduction or independently observed worker exhaustion. The relevant evidence is: - In [master QueryStatsUtil.getStats()](https://github.com/apache/doris/blob/87b462e4f58104140f098f5f4a8b5d58bec70149/fe/fe-core/src/main/java/org/apache/doris/statistics/query/QueryStatsUtil.java#L176-L249), normal completion, status rejection, empty results, missing-field validation exceptions, and RPC exceptions all leave the borrowed client outstanding. The [2.1.8-rc01 implementation](https://github.com/apache/doris/blob/2.1.8-rc01/fe/fe-core/src/main/java/org/apache/doris/statistics/query/QueryStatsUtil.java#L172-L242) has the same defect. - [ClientPool](https://github.com/apache/doris/blob/87b462e4f58104140f098f5f4a8b5d58bec70149/fe/fe-core/src/main/java/org/apache/doris/common/ClientPool.java#L43-L74) uses unlimited total and per-address active-client limits for `frontendPool`. Its 128 idle-client limit does not cap clients that were never returned. [GenericPool](https://github.com/apache/doris/blob/87b462e4f58104140f098f5f4a8b5d58bec70149/fe/fe-core/src/main/java/org/apache/doris/common/GenericPool.java#L152-L241) delegates ownership release to explicit return/invalidation; invalidation destroys the transport. - The reported worker symptom is consistent with the blocking [TThreadPoolServer setup](https://github.com/apache/doris/blob/87b462e4f58104140f098f5f4a8b5d58bec70149/fe/fe-core/src/main/java/org/apache/doris/common/ThriftServer.java#L143-L164). In [Config.java at this commit](https://github.com/apache/doris/blob/87b462e4f58104140f098f5f4a8b5d58bec70149/fe/fe-common/src/main/java/org/apache/doris/common/Config.java), `thrift_server_type` defaults to `THREAD_POOL`, and `thrift_client_timeout_ms` defaults to zero. Persistent connections can therefore retain workers waiting for their next request. Actual exhaustion depends on the deployed server type, timeout, traffic, and worker limit; it is not established merely by finding the leak. The next implementation step should be cleanup in a per-peer `finally` block, using the same address key as the borrow. Mark the RPC as completed immediately after `getQueryStats()` returns, before validating status or payload. Return the client on every completed RPC, including rejected status, empty payload, and subsequent local validation exceptions. Invalidate an acquired client if the RPC throws; do neither if no client was acquired. Preserve the existing result filtering and continuation to other peers. Suggested regression coverage: - Valid results, missing/non-OK status, empty payloads, and missing expected payload fields: exactly one return and no invalidation per acquired client. Cover each payload-validation branch. - RPC exception: exactly one invalidation and no return. Borrow failure: no cleanup of an unacquired client. Verify processing continues to the next live peer; self and dead peers are skipped. - Repeated sequential requests against two FEs: after warm-up, connections plateau instead of growing with request count, and outstanding borrows return to baseline after requests finish. Healthy idle pooled connections may remain open; expecting zero TCP connections would be incorrect. To tie the deployment symptoms to this verified defect, please add the exact FE version/build commit on each node, the precise SQL variant and request frequency/concurrency, FE topology, and effective `thrift_server_type`, `thrift_client_timeout_ms`, and `thrift_server_max_worker_threads`. Capture connection counts before/after a known number of requests, filtering both the peer RPC port and the requesting FE address—the supplied port-only command also counts unrelated clients. A peer FE thread dump showing the accumulating Thrift workers and corresponding FE timeout/rejection logs would establish the exhaustion mechanism. These details are not prerequisites for fixing the demonstrated ownership bug, but are needed to confirm the full operational impact and backport applicability. Until a verified fix is deployed, pause or reduce automated polling of the affected command to limit additional leaks. This does not release clients already leaked. Backport selection should follow verification against the exact target build; this investigation does not establish a fixed release. Breakwater-GitHub-Analysis-Slot: slot_274008bc5f42 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
