bzhang-obsec opened a new issue, #70632:
URL: https://github.com/apache/doris/issues/70632

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Version
   
   4.1.3 (cloud mode). The same code path is present in 4.1.5-release and 
master.
   
   ### What's Wrong?
   
   
   An OLAP `INSERT INTO ... SELECT` whose source the planner reduces to an 
empty set (for example, partition pruning on a
   LIST-partitioned table with no partition for the filtered value) leaves an 
`InsertLoadJob` in `PENDING` state in the master
   FE's `LoadManager` permanently. These jobs are never cleaned up and 
accumulate until the FE restarts.
   
   This is a path that #62282 (backport #62889) did not cover:
   
   1. For a normal OLAP insert, `InsertIntoTableCommand` builds 
`OlapInsertExecutor` with `needRegister = (jobId != -1)`
      
([InsertIntoTableCommand.java#L447-L448](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L447-L448)),
      and the constructor immediately calls `LoadManager.addLoadJob` with a new 
`InsertLoadJob` in `PENDING`
      
([AbstractInsertExecutor.java#L110-L112](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java#L110-L112)).
   2. If the sink's child is a `PhysicalEmptyRelation`, `runInternal` returns 
early, because `shouldExecuteEmptyInsert()` is
      `false` for OLAP tables
      
([InsertIntoTableCommand.java#L634-L638](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L634-L638)).
      `onComplete` / `afterExec` never run, so 
`OlapInsertExecutor.recordLoadJob` → `LoadManager.recordFinishedLoadJob`, the 
only
      transition to `FINISHED`/`CANCELLED`, is never called.
   3. The job can never be removed: `LoadJob.isExpired()` returns `false` for 
uncompleted jobs
      
([LoadJob.java#L1137-L1140](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java#L1137-L1140)),
      and `removeOverLimitLoadJob` only removes completed jobs. It is also 
never written to the edit log, so it disappears on an
      FE restart and nowhere else.
   
   On master the early return is `if (!insertExecutor.requiresTransaction()) 
return;`, with the same effect for OLAP tables.
   
   **Impact observed on a production cluster** (16 GiB master FE heap; a 
workload issuing many per-tenant `INSERT ... SELECT`
   statements that filter a LIST-partitioned table on a tenant that often has 
no partition):
   
   - `doris_fe_job{job="load", type="INSERT", state="PENDING"}` on the master 
grows by ~140k/day and only drops when the FE
     restarts (1.24M → 261 at one restart, then 1.97M after 13 days).
   - Master heap still in use after GC grows ~3.2 KB per stuck job (~0.44 
GiB/day). It reset together with the count at the
     restart, while the followers' heap stayed flat.
   - Once the heap fills, the master runs a stop-the-world full GC every ~5 
minutes, ~2.3 s each, stalling commits and DDL for
     the whole cluster.
   - Once more than `label_num_threshold` (2000) jobs are stuck, the hourly 
`LoadLabelCleaner` deletes **every** finished load
     job on each run (`removeOverLimitLoadJob` can never get below the 
threshold). `SHOW LOAD` therefore keeps less than an hour
     of history instead of `streaming_label_keep_max_second` / 
`label_keep_max_second`.
   - Every metrics scrape (`MetricRepo.getMetric` → 
`LoadManager.getLoadJobNum`) and every cleaner run walks all the stuck jobs.
   
   From the audit log, the stuck jobs match, hour by hour (98%), the INSERTs 
that return OK with 0 rows, 0 scanned rows and a
   query time of a few ms, and `EXPLAIN` of their SELECT shows a single 
`0:VEMPTYSET` node.
   
   
   ### What You Expected?
   
   An INSERT whose source is planned empty finishes its load job (as `FINISHED` 
with 0 rows, like an executed INSERT that
   writes nothing) or never registers one, so nothing is left in `LoadManager`.
   
   
   ### How to Reproduce?
   
   ```sql
   CREATE TABLE src (k VARCHAR(16) NOT NULL, v INT)
   DUPLICATE KEY(k)
   AUTO PARTITION BY LIST (k) ()
   DISTRIBUTED BY HASH(k) BUCKETS 1
   PROPERTIES ("replication_num" = "1");
   
   CREATE TABLE dst (k VARCHAR(16) NOT NULL, v INT)
   DUPLICATE KEY(k)
   AUTO PARTITION BY LIST (k) ()
   DISTRIBUTED BY HASH(k) BUCKETS 1
   PROPERTIES ("replication_num" = "1");
   
   INSERT INTO src VALUES ('a', 1);
   
   -- 'b' has no partition in src: the plan is a single VEMPTYSET
   EXPLAIN SELECT * FROM src WHERE k = 'b';
   
   -- returns OK, 0 rows; repeat it a few times
   INSERT INTO dst SELECT * FROM src WHERE k = 'b';
   ```
   
   Then on the master FE:
   
   ```
   curl -s http://<master-fe>:8030/metrics | grep 'doris_fe_job{job="load", 
type="INSERT", state="PENDING"}'
   ```
   
   The value goes up by one per INSERT and never goes down. It stays the same 
after `label_clean_interval_second`, and drops
   only when the FE restarts.
   
   ### Anything Else?
   
   Possible fixes, in order of preference:
   
   1. In the empty-source branch of `runInternal`, record the job as finished 
(0 rows) before returning, as `afterExec` does
      for an executed insert, so `SHOW LOAD` stays consistent.
   2. Alternatively, register the job only after the empty check, when the 
insert actually executes.
   
   As defense in depth, it may also be worth letting the label cleaner expire 
`PENDING` insert jobs older than the insert
   timeout. That would stop any future path like this from growing the heap 
without bound.
   
   Workaround on the client side: skip INSERTs whose `EXPLAIN` is a single 
`VEMPTYSET` node. They would write nothing anyway.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to