This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new de48fa225e4 [fix](fe) Finish the load job of an INSERT whose source is 
planned empty (#70649)
de48fa225e4 is described below

commit de48fa225e444ea4f25cf0699a656a901ad1ceb1
Author: Bei Zhang <[email protected]>
AuthorDate: Tue Nov 17 18:18:12 2026 -0800

    [fix](fe) Finish the load job of an INSERT whose source is planned empty 
(#70649)
    
    ### Background
    
    An OLAP `INSERT INTO ... SELECT` creates an `InsertLoadJob` and
    registers it with the master FE's `LoadManager` in PENDING when
    `OlapInsertExecutor` is constructed. The job becomes FINISHED or
    CANCELLED only through `LoadManager.recordFinishedLoadJob`, which
    `OlapInsertExecutor.afterExec` / `onFail` call once the insert has run.
    `SHOW LOAD` and the `doris_fe_job` metric read these jobs, and
    the label cleaner removes only completed ones.
    
    ### What problem does this PR solve?
    
    Issue Number: close #70632
    Related PR: #62282 (fixed the same leak for other insert paths), #62889
    (its backport)
    PR Type: Bugfix
    
    Problem Summary:
    
    If the planner reduces the source of an OLAP `INSERT INTO ... SELECT` to
    an empty set, the load job is registered but never finished. Two
    examples are partition pruning on a LIST-partitioned table that has no
    partition for the filtered value, and `WHERE 1 = 0`. The job stays
    PENDING in the master FE until it restarts, so a workload that often
    hits this leaks heap without bound. The fix records the registered job
    as FINISHED with 0 rows on that early-return path. After upgrading,
    these inserts show up in `SHOW LOAD` as FINISHED with 0 rows instead of
    PENDING.
    
    ### Details
    
    - **Frequency:** Every OLAP `INSERT ... SELECT` whose source is planned
    as a single `VEMPTYSET`, except `INSERT ... VALUES`, transaction
    mode and group commit, which do not register a job. The reporter saw
    about 140k stuck jobs per day on a production master. That filled a
    16 GiB heap and caused a stop-the-world full GC about every 5 minutes.
    Once more than `label_num_threshold` jobs were stuck, every run of
    the label cleaner also deleted all finished load jobs, so `SHOW LOAD`
    kept less than an hour of history.
    - **Root cause:** `InsertIntoTableCommand.runInternal` returns early
    when `insertExecutor.requiresTransaction()` is false, which is the
    case for an empty source without Table Stream offset updates. On that
    path neither `afterExec` nor `onFail` runs, so the job registered
    in the constructor is never finished. `LoadJob.isExpired()` is false for
    uncompleted jobs and `removeOverLimitLoadJob` removes only
    completed ones, so the job is never cleaned up. Registering a job
    doesn't write an edit-log entry, so it exists only in the master's
    memory
      and disappears on restart.
    - **Why existing tests missed it:** No test inserted from a source the
    planner reduces to an empty set and then checked the job's state.
    - **Fix:** Add `AbstractInsertExecutor.onSkipExecution()`, a hook that
    does nothing by default. `runInternal` calls it on the
    no-transaction early return. `OlapInsertExecutor` overrides it to call
    the same `recordLoadJob` that `afterExec` uses, but only when its
    constructor registered the job. No transaction is faked: the job is
    recorded with transaction id `-1`, and the client response is
      unchanged.
    - **Other paths and branches:** `OlapTxnInsertExecutor`,
    `OlapGroupCommitInsertExecutor`, `RemoteOlapInsertExecutor` and the
    external-table executors don't register a job, so for them the hook does
    nothing. The empty-source paths in `RewriteTableCommand` and
    `RowLevelDmlCommand` don't use a registering executor. The same leak
    exists on branch-4.1 (4.1.3 and 4.1.5), where the early return
      checks `shouldExecuteEmptyInsert()`, so this fix should be backported.
    - **Behavior change:** These inserts now appear in `SHOW LOAD` as
    FINISHED with 0 rows. Previously they showed
    as PENDING until the FE restarted. Each one now also writes one
    `logCreateLoadJob` edit-log entry, the same as an executed insert.
    
    ### Testing
    
    - `OlapInsertExecutorTest`: `testSkipExecutionFinishesRegisteredLoadJob`
    checks that a registered empty insert calls
    `recordFinishedLoadJob` with no fail message, which makes the job
    FINISHED. `testSkipExecutionRecordsNothingForUnregisteredLoadJob` checks
    that an executor which didn't register a job doesn't touch
    `LoadManager`.
    - `load_p0/insert/test_insert_statistic`: runs the issue's repro. The
    source is an auto LIST-partitioned table filtered on a value with no
    partition, and the test asserts its plan contains `VEMPTYSET`. It also
    runs a `WHERE 1 = 0` source. For each labelled insert it asserts
    `SHOW LOAD` reports FINISHED, then asserts no job in the database is
    PENDING. Before this fix both labels report PENDING.
    
    None of these were run locally. They are left to CI.
    
    ### Notes for Reviewers
    
    I left the issue's suggested safeguard, expiring old PENDING insert jobs
    in the label cleaner, out of this PR. A live insert is also
    PENDING until it completes, so a job's age alone doesn't show that it's
    orphaned.
    
    ### Release note
    
    Fix `INSERT INTO ... SELECT` statements whose source is planned empty,
    such as one filtered on a partition that doesn't exist, leaving
    PENDING load jobs in the master FE that leaked memory until restart.
    
    ### Check List (For Author)
    
    - Behavior changed: Yes (see Details)
    - Does this need documentation: No
    - Pre-submission status: Partially verified: the new unit test and
    regression case have not been run locally; they rely on CI.
    
    ### AI Disclosure
    
    - Tool: Claude Code (Claude Opus 5.5)
    - Generated: Code, tests, and this description
    - Human review: All of it
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
 .../commands/insert/AbstractInsertExecutor.java    |  9 +++++
 .../commands/insert/InsertIntoTableCommand.java    |  1 +
 .../plans/commands/insert/OlapInsertExecutor.java  |  7 ++++
 .../commands/insert/OlapInsertExecutorTest.java    | 47 +++++++++++++++++++++-
 .../load_p0/insert/test_insert_statistic.groovy    | 32 +++++++++++++++
 5 files changed, 95 insertions(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
index c57b97ae39e..cee64184057 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
@@ -81,6 +81,7 @@ public abstract class AbstractInsertExecutor {
     protected String errMsg = "";
     protected Optional<InsertCommandContext> insertCtx;
     protected final boolean emptyInsert;
+    protected final boolean loadJobRegistered;
     protected long txnId = INVALID_TXN_ID;
     protected List<TableStreamUpdateInfo> streamUpdateInfos = 
Collections.emptyList();
 
@@ -138,6 +139,7 @@ public abstract class AbstractInsertExecutor {
         this.insertCtx = insertCtx;
         this.emptyInsert = emptyInsert;
         this.jobId = jobId;
+        this.loadJobRegistered = needRegister;
     }
 
     public void registerListener(InsertExecutorListener listener) {
@@ -202,6 +204,13 @@ public abstract class AbstractInsertExecutor {
      */
     protected abstract void afterExec(StmtExecutor executor);
 
+    /**
+     * Called instead of executeSingleInsert when the insert needs no 
transaction, i.e. the planner proved the
+     * source empty. Must close any bookkeeping opened in the constructor, 
such as a registered load job.
+     */
+    public void onSkipExecution() {
+    }
+
     protected final void execImpl(StmtExecutor executor) throws Exception {
         String queryId = DebugUtil.printId(ctx.queryId());
         
coordinator.setLoadZeroTolerance(ctx.getSessionVariable().getEnableInsertStrict());
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
index 5cd535122c8..bc07f06b4d5 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
@@ -781,6 +781,7 @@ public class InsertIntoTableCommand extends Command
             // they are not -- no row and no offset was written, which is what 
it needs before treating a
             // cancellation or a later failure here as too late to take back.
             // See InsertCommandContext#setCommitted.
+            insertExecutor.onSkipExecution();
             return;
         }
         if (insertExecutorListener != null) {
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
index 70387f50619..9b980d5aadf 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
@@ -364,6 +364,13 @@ public class OlapInsertExecutor extends 
AbstractInsertExecutor {
         setReturnInfo();
     }
 
+    @Override
+    public void onSkipExecution() {
+        if (loadJobRegistered) {
+            recordLoadJob(ctx.getCurrentUserIdentity());
+        }
+    }
+
     protected void setReturnInfo() {
         // {'label':'my_label1', 'status':'visible', 'txnId':'123'}
         // {'label':'my_label1', 'status':'visible', 'txnId':'123' 
'err':'error messages'}
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
index 793b9d60fa6..45923c5ffee 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
@@ -119,6 +119,47 @@ class OlapInsertExecutorTest {
         }
     }
 
+    @Test
+    void testSkipExecutionFinishesRegisteredLoadJob() throws Exception {
+        ConnectContext ctx = createExecutorContext();
+        LoadManager loadManager = Mockito.mock(LoadManager.class);
+        Env currentEnv = createCurrentEnv(loadManager);
+
+        try (MockedStatic<EnvFactory> envFactoryMock = 
Mockito.mockStatic(EnvFactory.class);
+                MockedStatic<Env> envMock = Mockito.mockStatic(Env.class)) {
+            prepareFactoryMocks(envFactoryMock, envMock, createCoordinator(),
+                    Mockito.mock(GlobalTransactionMgrIface.class), 
Mockito.mock(TransactionState.class), currentEnv);
+            ctx.setEnv(currentEnv);
+
+            OlapInsertExecutor executor = createExecutor(ctx, true, true);
+            Mockito.verify(loadManager).addLoadJob(Mockito.any());
+            executor.onSkipExecution();
+
+            
Mockito.verify(loadManager).recordFinishedLoadJob(Mockito.eq("label_test"), 
Mockito.eq(-1L),
+                    Mockito.eq("test_db"), Mockito.eq(2L), 
Mockito.eq(EtlJobType.INSERT), Mockito.anyLong(),
+                    Mockito.eq(""), Mockito.isNull(), Mockito.isNull(), 
Mockito.eq(UserIdentity.ROOT),
+                    Mockito.anyLong());
+        }
+    }
+
+    @Test
+    void testSkipExecutionRecordsNothingForUnregisteredLoadJob() throws 
Exception {
+        ConnectContext ctx = createExecutorContext();
+        LoadManager loadManager = Mockito.mock(LoadManager.class);
+        Env currentEnv = createCurrentEnv(loadManager);
+
+        try (MockedStatic<EnvFactory> envFactoryMock = 
Mockito.mockStatic(EnvFactory.class);
+                MockedStatic<Env> envMock = Mockito.mockStatic(Env.class)) {
+            prepareFactoryMocks(envFactoryMock, envMock, createCoordinator(),
+                    Mockito.mock(GlobalTransactionMgrIface.class), 
Mockito.mock(TransactionState.class), currentEnv);
+            ctx.setEnv(currentEnv);
+
+            createExecutor(ctx, true, false).onSkipExecution();
+
+            Mockito.verifyNoInteractions(loadManager);
+        }
+    }
+
     @Test
     void testPublishTimeoutCommittedModeReturnsOk() throws Exception {
         ConnectContext ctx = createExecutorContext();
@@ -412,6 +453,10 @@ class OlapInsertExecutorTest {
     }
 
     private OlapInsertExecutor createExecutor(ConnectContext ctx, boolean 
emptyInsert) {
+        return createExecutor(ctx, emptyInsert, false);
+    }
+
+    private OlapInsertExecutor createExecutor(ConnectContext ctx, boolean 
emptyInsert, boolean needRegister) {
         Database database = Mockito.mock(Database.class);
         Mockito.when(database.getFullName()).thenReturn("test_db");
         Mockito.when(database.getId()).thenReturn(1L);
@@ -423,7 +468,7 @@ class OlapInsertExecutorTest {
         Mockito.when(table.getId()).thenReturn(2L);
 
         return new OlapInsertExecutor(ctx, table, "label_test", 
Mockito.mock(NereidsPlanner.class),
-                Optional.empty(), emptyInsert, 0L);
+                Optional.empty(), emptyInsert, 0L, needRegister);
     }
 
     private OlapInsertExecutor 
createExecutorWithBeforeExecFailure(ConnectContext ctx) {
diff --git a/regression-test/suites/load_p0/insert/test_insert_statistic.groovy 
b/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
index ffc1263b5f4..4c152c16017 100644
--- a/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
+++ b/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
@@ -109,6 +109,38 @@ suite("test_insert_statistic", "p0") {
     def cancelledJob = result.find { it[2] == "CANCELLED" }
     assertNotNull(cancelledJob, "Expected a CANCELLED load job for failed 
insert")
 
+    // insert into select whose source is planned as an empty set → job should 
be FINISHED, not left PENDING
+    sql """ DROP TABLE IF EXISTS ${insert_tbl}_empty_src"""
+    sql """
+     CREATE TABLE ${insert_tbl}_empty_src (
+       `k1` varchar(16) NOT NULL,
+       `k2` int NULL
+     ) ENGINE=OLAP
+     DUPLICATE KEY(`k1`)
+     AUTO PARTITION BY LIST (`k1`) ()
+     DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+     PROPERTIES ("replication_num"="1");
+    """
+    sql """ INSERT INTO ${insert_tbl}_empty_src VALUES ('a', 1) """
+    explain {
+        sql """ SELECT * FROM ${insert_tbl}_empty_src WHERE k1 = 'b' """
+        contains "VEMPTYSET"
+    }
+    def emptyLabels = []
+    ["k1 = 'b'", "1 = 0"].each { predicate ->
+        def emptyLabel = "label_insert_empty_select_" + emptyLabels.size() + 
"_" + System.currentTimeMillis()
+        sql """ INSERT INTO ${insert_tbl}_2 WITH LABEL ${emptyLabel}
+                SELECT k2, k2, k2, k2 FROM ${insert_tbl}_empty_src WHERE 
${predicate} """
+        emptyLabels.add(emptyLabel)
+    }
+    emptyLabels.each { emptyLabel ->
+        result = sql "SHOW LOAD FROM ${dbName} WHERE LABEL = '${emptyLabel}'"
+        assertEquals(1, result.size())
+        assertEquals("FINISHED", result[0][2])
+    }
+    result = sql "SHOW LOAD FROM ${dbName}"
+    assertNull(result.find { it[2] == "PENDING" }, "No insert load job should 
be left PENDING")
+
     // insert into s3 tvf
     String ak = getS3AK()
     String sk = getS3SK()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to