Doris-Breakwater commented on issue #70632: URL: https://github.com/apache/doris/issues/70632#issuecomment-6914884798
Breakwater-GitHub-Analysis-Slot: slot_a4f61110efc2 **Assessment:** The reported path is supported by source inspection: a normal OLAP insert can register an `InsertLoadJob` and then return successfully for a planner-proven empty source without completing that job. This is a remaining lifecycle gap after [#62282](https://github.com/apache/doris/pull/62282) / [#62889](https://github.com/apache/doris/pull/62889). Suggested triage: FE / INSERT / memory leak, with high priority given the reported master-FE availability impact. The issue currently has no labels or assignee. **Verified in code** I checked the configured local repository at `4.1.3` (commit `7126cf65d96e`) and `4.1.5-release` (`917b20be8387`), and fetched the relevant upstream master files at `13cff7ff484596f29aa1381750e906db3c8bb704` (2026-11-16). This was static investigation; I did not run the SQL against a cluster or modify source code. - In 4.1.3, ordinary OLAP INSERT SELECT passes `needRegister = jobId != -1`; the base constructor immediately registers the job. `LoadJob.state` initially equals `PENDING`. Registration retains it in both the ID map and the database/label index. See [executor selection](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L430-L448), [constructor](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java#L106-L120), and [registration/completion](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java#L190-L235). - For the empty OLAP path, `initPlan` skips transaction creation, and `runInternal` returns before either execution method. OLAP inherits `shouldExecuteEmptyInsert() == false`. Consequently, neither successful completion bookkeeping nor failure bookkeeping runs. The 4.1.5-release path has the same gap. See [4.1.3 transaction gate](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L306-L314) and [4.1.5 early return](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L660-L677). - Current master still registers before [the `requiresTransaction()` return](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L771-L786). For an ordinary empty OLAP source without stream offset updates, this predicate is false. - Normal retention cannot reclaim these entries: [`isExpired`](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java#L1137-L1151) rejects unfinished jobs, while [count-based cleanup](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java#L417-L434) selects only completed jobs. If stuck entries alone exceed the threshold, that loop exhausts all completed jobs present in this manager. Metrics aggregation also traverses the retained jobs. Registration does not journal the job; completion does, which is consistent with the reported master-only growth and restart reset. The production growth rate, retained bytes per job, GC pauses, and audit correlation remain reporter-provided observations, not independently measured results. The mechanism is sufficient for triage, but does not prove that every pending job or every pause has this cause. The problematic boundary is the physical empty plan, rather than LIST partitioning specifically. An executed query that happens to return zero rows follows a different lifecycle. **Recommended fix and validation** 1. Prefer an OLAP-specific completion path for the successful no-op that finalizes an already registered job as `FINISHED`, with zero-row statistics and normal terminal timestamps/persistence. Alternatively, suppress registration for inserts that will take the no-transaction return, if omitting their SHOW LOAD history is accepted. 2. Do not simply invoke the existing OLAP `onComplete()` or `afterExec()` unchanged: no transaction was opened (`txnId == -1`), `onComplete()` expects a real transaction, and the executor's initial transaction status is `ABORTED`. Preserve a correct successful client response without fabricating a committed transaction. Keep intentional non-registration for VALUES, internal group commit, and external executors. 3. Preserve empty-source side effects: 4.1.x has external overwrite handling, and master [requires a transaction for stream offset updates even with no output rows](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java#L350-L357). A blanket empty-source skip would break those semantics. 4. Extend `regression-test/suites/load_p0/insert/test_insert_statistic.groovy` with repeated missing-LIST-partition inserts and a constant-false source. Assert the INSERT plan is empty and each uniquely labeled job becomes terminal (or is absent under the alternate design), with no residual PENDING growth. Cover an executed zero-row query, ordinary success/failure, and the existing empty-source side-effect cases. Validate cleanup and, if recording finished jobs, replay after restart. Backport after testing each affected branch. Treat an age-based PENDING cleaner as a separate change. Live INSERT jobs also remain PENDING until completion, so age alone does not establish that an entry is orphaned. Lowering retention or increasing the count threshold will not remove the stuck entries. **Remaining evidence / next steps** The supplied reproducer is enough to start a regression and PR; a full production dump is not a prerequisite. Please add the exact FE build hash, relevant session settings (especially group commit), the full `EXPLAIN INSERT INTO ... SELECT ...` output, and uniquely labeled SHOW LOAD results plus before/after master PENDING counts from an otherwise quiet reproduction. These confirm that the actual INSERT selects the claimed executor and physical plan. A redacted heap dominator/class-histogram comparison and time-aligned GC logs would help quantify retained memory and pause attribution if needed. For temporary prevention, skipping a plain INSERT whose source is authoritatively known to be empty avoids this path. A separate EXPLAIN can become stale when source data or partitions change, so it should not become a general correctness rule. It also does not reclaim existing entries. The reporter's willingness to submit a PR is appropriate; the missing regression should accompany it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
