This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new de48fa225e4 [fix](fe) Finish the load job of an INSERT whose source is
planned empty (#70649)
de48fa225e4 is described below
commit de48fa225e444ea4f25cf0699a656a901ad1ceb1
Author: Bei Zhang <[email protected]>
AuthorDate: Tue Nov 17 18:18:12 2026 -0800
[fix](fe) Finish the load job of an INSERT whose source is planned empty
(#70649)
### Background
An OLAP `INSERT INTO ... SELECT` creates an `InsertLoadJob` and
registers it with the master FE's `LoadManager` in PENDING when
`OlapInsertExecutor` is constructed. The job becomes FINISHED or
CANCELLED only through `LoadManager.recordFinishedLoadJob`, which
`OlapInsertExecutor.afterExec` / `onFail` call once the insert has run.
`SHOW LOAD` and the `doris_fe_job` metric read these jobs, and
the label cleaner removes only completed ones.
### What problem does this PR solve?
Issue Number: close #70632
Related PR: #62282 (fixed the same leak for other insert paths), #62889
(its backport)
PR Type: Bugfix
Problem Summary:
If the planner reduces the source of an OLAP `INSERT INTO ... SELECT` to
an empty set, the load job is registered but never finished. Two
examples are partition pruning on a LIST-partitioned table that has no
partition for the filtered value, and `WHERE 1 = 0`. The job stays
PENDING in the master FE until it restarts, so a workload that often
hits this leaks heap without bound. The fix records the registered job
as FINISHED with 0 rows on that early-return path. After upgrading,
these inserts show up in `SHOW LOAD` as FINISHED with 0 rows instead of
PENDING.
### Details
- **Frequency:** Every OLAP `INSERT ... SELECT` whose source is planned
as a single `VEMPTYSET`, except `INSERT ... VALUES`, transaction
mode and group commit, which do not register a job. The reporter saw
about 140k stuck jobs per day on a production master. That filled a
16 GiB heap and caused a stop-the-world full GC about every 5 minutes.
Once more than `label_num_threshold` jobs were stuck, every run of
the label cleaner also deleted all finished load jobs, so `SHOW LOAD`
kept less than an hour of history.
- **Root cause:** `InsertIntoTableCommand.runInternal` returns early
when `insertExecutor.requiresTransaction()` is false, which is the
case for an empty source without Table Stream offset updates. On that
path neither `afterExec` nor `onFail` runs, so the job registered
in the constructor is never finished. `LoadJob.isExpired()` is false for
uncompleted jobs and `removeOverLimitLoadJob` removes only
completed ones, so the job is never cleaned up. Registering a job
doesn't write an edit-log entry, so it exists only in the master's
memory
and disappears on restart.
- **Why existing tests missed it:** No test inserted from a source the
planner reduces to an empty set and then checked the job's state.
- **Fix:** Add `AbstractInsertExecutor.onSkipExecution()`, a hook that
does nothing by default. `runInternal` calls it on the
no-transaction early return. `OlapInsertExecutor` overrides it to call
the same `recordLoadJob` that `afterExec` uses, but only when its
constructor registered the job. No transaction is faked: the job is
recorded with transaction id `-1`, and the client response is
unchanged.
- **Other paths and branches:** `OlapTxnInsertExecutor`,
`OlapGroupCommitInsertExecutor`, `RemoteOlapInsertExecutor` and the
external-table executors don't register a job, so for them the hook does
nothing. The empty-source paths in `RewriteTableCommand` and
`RowLevelDmlCommand` don't use a registering executor. The same leak
exists on branch-4.1 (4.1.3 and 4.1.5), where the early return
checks `shouldExecuteEmptyInsert()`, so this fix should be backported.
- **Behavior change:** These inserts now appear in `SHOW LOAD` as
FINISHED with 0 rows. Previously they showed
as PENDING until the FE restarted. Each one now also writes one
`logCreateLoadJob` edit-log entry, the same as an executed insert.
### Testing
- `OlapInsertExecutorTest`: `testSkipExecutionFinishesRegisteredLoadJob`
checks that a registered empty insert calls
`recordFinishedLoadJob` with no fail message, which makes the job
FINISHED. `testSkipExecutionRecordsNothingForUnregisteredLoadJob` checks
that an executor which didn't register a job doesn't touch
`LoadManager`.
- `load_p0/insert/test_insert_statistic`: runs the issue's repro. The
source is an auto LIST-partitioned table filtered on a value with no
partition, and the test asserts its plan contains `VEMPTYSET`. It also
runs a `WHERE 1 = 0` source. For each labelled insert it asserts
`SHOW LOAD` reports FINISHED, then asserts no job in the database is
PENDING. Before this fix both labels report PENDING.
None of these were run locally. They are left to CI.
### Notes for Reviewers
I left the issue's suggested safeguard, expiring old PENDING insert jobs
in the label cleaner, out of this PR. A live insert is also
PENDING until it completes, so a job's age alone doesn't show that it's
orphaned.
### Release note
Fix `INSERT INTO ... SELECT` statements whose source is planned empty,
such as one filtered on a partition that doesn't exist, leaving
PENDING load jobs in the master FE that leaked memory until restart.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the new unit test and
regression case have not been run locally; they rely on CI.
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: Code, tests, and this description
- Human review: All of it
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
.../commands/insert/AbstractInsertExecutor.java | 9 +++++
.../commands/insert/InsertIntoTableCommand.java | 1 +
.../plans/commands/insert/OlapInsertExecutor.java | 7 ++++
.../commands/insert/OlapInsertExecutorTest.java | 47 +++++++++++++++++++++-
.../load_p0/insert/test_insert_statistic.groovy | 32 +++++++++++++++
5 files changed, 95 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
index c57b97ae39e..cee64184057 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
@@ -81,6 +81,7 @@ public abstract class AbstractInsertExecutor {
protected String errMsg = "";
protected Optional<InsertCommandContext> insertCtx;
protected final boolean emptyInsert;
+ protected final boolean loadJobRegistered;
protected long txnId = INVALID_TXN_ID;
protected List<TableStreamUpdateInfo> streamUpdateInfos =
Collections.emptyList();
@@ -138,6 +139,7 @@ public abstract class AbstractInsertExecutor {
this.insertCtx = insertCtx;
this.emptyInsert = emptyInsert;
this.jobId = jobId;
+ this.loadJobRegistered = needRegister;
}
public void registerListener(InsertExecutorListener listener) {
@@ -202,6 +204,13 @@ public abstract class AbstractInsertExecutor {
*/
protected abstract void afterExec(StmtExecutor executor);
+ /**
+ * Called instead of executeSingleInsert when the insert needs no
transaction, i.e. the planner proved the
+ * source empty. Must close any bookkeeping opened in the constructor,
such as a registered load job.
+ */
+ public void onSkipExecution() {
+ }
+
protected final void execImpl(StmtExecutor executor) throws Exception {
String queryId = DebugUtil.printId(ctx.queryId());
coordinator.setLoadZeroTolerance(ctx.getSessionVariable().getEnableInsertStrict());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
index 5cd535122c8..bc07f06b4d5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java
@@ -781,6 +781,7 @@ public class InsertIntoTableCommand extends Command
// they are not -- no row and no offset was written, which is what
it needs before treating a
// cancellation or a later failure here as too late to take back.
// See InsertCommandContext#setCommitted.
+ insertExecutor.onSkipExecution();
return;
}
if (insertExecutorListener != null) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
index 70387f50619..9b980d5aadf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java
@@ -364,6 +364,13 @@ public class OlapInsertExecutor extends
AbstractInsertExecutor {
setReturnInfo();
}
+ @Override
+ public void onSkipExecution() {
+ if (loadJobRegistered) {
+ recordLoadJob(ctx.getCurrentUserIdentity());
+ }
+ }
+
protected void setReturnInfo() {
// {'label':'my_label1', 'status':'visible', 'txnId':'123'}
// {'label':'my_label1', 'status':'visible', 'txnId':'123'
'err':'error messages'}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
index 793b9d60fa6..45923c5ffee 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutorTest.java
@@ -119,6 +119,47 @@ class OlapInsertExecutorTest {
}
}
+ @Test
+ void testSkipExecutionFinishesRegisteredLoadJob() throws Exception {
+ ConnectContext ctx = createExecutorContext();
+ LoadManager loadManager = Mockito.mock(LoadManager.class);
+ Env currentEnv = createCurrentEnv(loadManager);
+
+ try (MockedStatic<EnvFactory> envFactoryMock =
Mockito.mockStatic(EnvFactory.class);
+ MockedStatic<Env> envMock = Mockito.mockStatic(Env.class)) {
+ prepareFactoryMocks(envFactoryMock, envMock, createCoordinator(),
+ Mockito.mock(GlobalTransactionMgrIface.class),
Mockito.mock(TransactionState.class), currentEnv);
+ ctx.setEnv(currentEnv);
+
+ OlapInsertExecutor executor = createExecutor(ctx, true, true);
+ Mockito.verify(loadManager).addLoadJob(Mockito.any());
+ executor.onSkipExecution();
+
+
Mockito.verify(loadManager).recordFinishedLoadJob(Mockito.eq("label_test"),
Mockito.eq(-1L),
+ Mockito.eq("test_db"), Mockito.eq(2L),
Mockito.eq(EtlJobType.INSERT), Mockito.anyLong(),
+ Mockito.eq(""), Mockito.isNull(), Mockito.isNull(),
Mockito.eq(UserIdentity.ROOT),
+ Mockito.anyLong());
+ }
+ }
+
+ @Test
+ void testSkipExecutionRecordsNothingForUnregisteredLoadJob() throws
Exception {
+ ConnectContext ctx = createExecutorContext();
+ LoadManager loadManager = Mockito.mock(LoadManager.class);
+ Env currentEnv = createCurrentEnv(loadManager);
+
+ try (MockedStatic<EnvFactory> envFactoryMock =
Mockito.mockStatic(EnvFactory.class);
+ MockedStatic<Env> envMock = Mockito.mockStatic(Env.class)) {
+ prepareFactoryMocks(envFactoryMock, envMock, createCoordinator(),
+ Mockito.mock(GlobalTransactionMgrIface.class),
Mockito.mock(TransactionState.class), currentEnv);
+ ctx.setEnv(currentEnv);
+
+ createExecutor(ctx, true, false).onSkipExecution();
+
+ Mockito.verifyNoInteractions(loadManager);
+ }
+ }
+
@Test
void testPublishTimeoutCommittedModeReturnsOk() throws Exception {
ConnectContext ctx = createExecutorContext();
@@ -412,6 +453,10 @@ class OlapInsertExecutorTest {
}
private OlapInsertExecutor createExecutor(ConnectContext ctx, boolean
emptyInsert) {
+ return createExecutor(ctx, emptyInsert, false);
+ }
+
+ private OlapInsertExecutor createExecutor(ConnectContext ctx, boolean
emptyInsert, boolean needRegister) {
Database database = Mockito.mock(Database.class);
Mockito.when(database.getFullName()).thenReturn("test_db");
Mockito.when(database.getId()).thenReturn(1L);
@@ -423,7 +468,7 @@ class OlapInsertExecutorTest {
Mockito.when(table.getId()).thenReturn(2L);
return new OlapInsertExecutor(ctx, table, "label_test",
Mockito.mock(NereidsPlanner.class),
- Optional.empty(), emptyInsert, 0L);
+ Optional.empty(), emptyInsert, 0L, needRegister);
}
private OlapInsertExecutor
createExecutorWithBeforeExecFailure(ConnectContext ctx) {
diff --git a/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
b/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
index ffc1263b5f4..4c152c16017 100644
--- a/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
+++ b/regression-test/suites/load_p0/insert/test_insert_statistic.groovy
@@ -109,6 +109,38 @@ suite("test_insert_statistic", "p0") {
def cancelledJob = result.find { it[2] == "CANCELLED" }
assertNotNull(cancelledJob, "Expected a CANCELLED load job for failed
insert")
+ // insert into select whose source is planned as an empty set → job should
be FINISHED, not left PENDING
+ sql """ DROP TABLE IF EXISTS ${insert_tbl}_empty_src"""
+ sql """
+ CREATE TABLE ${insert_tbl}_empty_src (
+ `k1` varchar(16) NOT NULL,
+ `k2` int NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ AUTO PARTITION BY LIST (`k1`) ()
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ PROPERTIES ("replication_num"="1");
+ """
+ sql """ INSERT INTO ${insert_tbl}_empty_src VALUES ('a', 1) """
+ explain {
+ sql """ SELECT * FROM ${insert_tbl}_empty_src WHERE k1 = 'b' """
+ contains "VEMPTYSET"
+ }
+ def emptyLabels = []
+ ["k1 = 'b'", "1 = 0"].each { predicate ->
+ def emptyLabel = "label_insert_empty_select_" + emptyLabels.size() +
"_" + System.currentTimeMillis()
+ sql """ INSERT INTO ${insert_tbl}_2 WITH LABEL ${emptyLabel}
+ SELECT k2, k2, k2, k2 FROM ${insert_tbl}_empty_src WHERE
${predicate} """
+ emptyLabels.add(emptyLabel)
+ }
+ emptyLabels.each { emptyLabel ->
+ result = sql "SHOW LOAD FROM ${dbName} WHERE LABEL = '${emptyLabel}'"
+ assertEquals(1, result.size())
+ assertEquals("FINISHED", result[0][2])
+ }
+ result = sql "SHOW LOAD FROM ${dbName}"
+ assertNull(result.find { it[2] == "PENDING" }, "No insert load job should
be left PENDING")
+
// insert into s3 tvf
String ak = getS3AK()
String sk = getS3SK()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]