bzhang-obsec opened a new issue, #70632: URL: https://github.com/apache/doris/issues/70632
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.3 (cloud mode). The same code path is present in 4.1.5-release and master. ### What's Wrong? An OLAP `INSERT INTO ... SELECT` whose source the planner reduces to an empty set (for example, partition pruning on a LIST-partitioned table with no partition for the filtered value) leaves an `InsertLoadJob` in `PENDING` state in the master FE's `LoadManager` permanently. These jobs are never cleaned up and accumulate until the FE restarts. This is a path that #62282 (backport #62889) did not cover: 1. For a normal OLAP insert, `InsertIntoTableCommand` builds `OlapInsertExecutor` with `needRegister = (jobId != -1)` ([InsertIntoTableCommand.java#L447-L448](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L447-L448)), and the constructor immediately calls `LoadManager.addLoadJob` with a new `InsertLoadJob` in `PENDING` ([AbstractInsertExecutor.java#L110-L112](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java#L110-L112)). 2. If the sink's child is a `PhysicalEmptyRelation`, `runInternal` returns early, because `shouldExecuteEmptyInsert()` is `false` for OLAP tables ([InsertIntoTableCommand.java#L634-L638](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L634-L638)). `onComplete` / `afterExec` never run, so `OlapInsertExecutor.recordLoadJob` → `LoadManager.recordFinishedLoadJob`, the only transition to `FINISHED`/`CANCELLED`, is never called. 3. The job can never be removed: `LoadJob.isExpired()` returns `false` for uncompleted jobs ([LoadJob.java#L1137-L1140](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java#L1137-L1140)), and `removeOverLimitLoadJob` only removes completed jobs. It is also never written to the edit log, so it disappears on an FE restart and nowhere else. On master the early return is `if (!insertExecutor.requiresTransaction()) return;`, with the same effect for OLAP tables. **Impact observed on a production cluster** (16 GiB master FE heap; a workload issuing many per-tenant `INSERT ... SELECT` statements that filter a LIST-partitioned table on a tenant that often has no partition): - `doris_fe_job{job="load", type="INSERT", state="PENDING"}` on the master grows by ~140k/day and only drops when the FE restarts (1.24M → 261 at one restart, then 1.97M after 13 days). - Master heap still in use after GC grows ~3.2 KB per stuck job (~0.44 GiB/day). It reset together with the count at the restart, while the followers' heap stayed flat. - Once the heap fills, the master runs a stop-the-world full GC every ~5 minutes, ~2.3 s each, stalling commits and DDL for the whole cluster. - Once more than `label_num_threshold` (2000) jobs are stuck, the hourly `LoadLabelCleaner` deletes **every** finished load job on each run (`removeOverLimitLoadJob` can never get below the threshold). `SHOW LOAD` therefore keeps less than an hour of history instead of `streaming_label_keep_max_second` / `label_keep_max_second`. - Every metrics scrape (`MetricRepo.getMetric` → `LoadManager.getLoadJobNum`) and every cleaner run walks all the stuck jobs. From the audit log, the stuck jobs match, hour by hour (98%), the INSERTs that return OK with 0 rows, 0 scanned rows and a query time of a few ms, and `EXPLAIN` of their SELECT shows a single `0:VEMPTYSET` node. ### What You Expected? An INSERT whose source is planned empty finishes its load job (as `FINISHED` with 0 rows, like an executed INSERT that writes nothing) or never registers one, so nothing is left in `LoadManager`. ### How to Reproduce? ```sql CREATE TABLE src (k VARCHAR(16) NOT NULL, v INT) DUPLICATE KEY(k) AUTO PARTITION BY LIST (k) () DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1"); CREATE TABLE dst (k VARCHAR(16) NOT NULL, v INT) DUPLICATE KEY(k) AUTO PARTITION BY LIST (k) () DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1"); INSERT INTO src VALUES ('a', 1); -- 'b' has no partition in src: the plan is a single VEMPTYSET EXPLAIN SELECT * FROM src WHERE k = 'b'; -- returns OK, 0 rows; repeat it a few times INSERT INTO dst SELECT * FROM src WHERE k = 'b'; ``` Then on the master FE: ``` curl -s http://<master-fe>:8030/metrics | grep 'doris_fe_job{job="load", type="INSERT", state="PENDING"}' ``` The value goes up by one per INSERT and never goes down. It stays the same after `label_clean_interval_second`, and drops only when the FE restarts. ### Anything Else? Possible fixes, in order of preference: 1. In the empty-source branch of `runInternal`, record the job as finished (0 rows) before returning, as `afterExec` does for an executed insert, so `SHOW LOAD` stays consistent. 2. Alternatively, register the job only after the empty check, when the insert actually executes. As defense in depth, it may also be worth letting the label cleaner expire `PENDING` insert jobs older than the insert timeout. That would stop any future path like this from growing the heap without bound. Workaround on the client side: skip INSERTs whose `EXPLAIN` is a single `VEMPTYSET` node. They would write nothing anyway. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
