sollhui opened a new pull request, #62282: URL: https://github.com/apache/doris/pull/62282
fix https://github.com/apache/doris/issues/62118 ## Problem When users use HTTP StreamLoad with `group_commit=async_mode`, the BE internally creates group commit batches via `GroupCommitTable::_create_group_commit_load`, which sends a `streamLoadPut` RPC to FE with a SQL like: INSERT INTO doris_internal_table_id(table_id) WITH LABEL group_commit_xxx SELECT * FROM group_commit("table_id"="...") FE processes this via `httpStreamPutImpl`, which calls `initPlan()` and registers an `InsertLoadJob` in `LoadManager` with PENDING state. However, this job **never transitions to FINISHED/CANCELLED** because: 1. `httpStreamPutImpl` only generates the plan and returns it to BE — it never calls `executeSingleInsert()`. 2. The actual data commit is done by BE via `loadTxnCommit` RPC, which bypasses `LoadManager` entirely. 3. `isExpired()` returns false for non-completed jobs, so these PENDING jobs are never cleaned up. This causes a steady memory leak in `LoadManager`. The issue was introduced by #56412 and was partially fixed by #56852 (which only handled the case where `ctx.isGroupCommit() == true`). The internal group commit RPC does not set `group_commit_mode`, so `ctx.isGroupCommit()` remains false and the fix in #56852 does not apply. ## Fix In `InsertIntoTableCommand.selectInsertExecutorFactory`, detect when the INSERT source contains a `group_commit(...)` TVF and set `jobId = -1`, preventing `addLoadJob` from being called. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
