freemandealer opened a new pull request, #70282:
URL: https://github.com/apache/doris/pull/70282
### Background
Scheduled INSERT jobs execute SQL through `InsertTask` and persist their
task history. `INSERT INTO s3(...) SELECT ...` already supports external
output, but job creation accepts only table INSERT commands.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Enhancement
Problem Summary:
Allow an existing S3 INSERT to run as an immediate, one-time job. Validate
its target, schedule, properties and query before registering the job, without
deleting or writing destination files during validation. Route execution
through `InsertTask`, handle cancellation and failures consistently, and mask
TVF credentials in audit and job SQL displays. Most of the diff consists of
tests for creation, execution, cancellation races, persistence and table-job
compatibility.
### Details
- **Change map:** 394 changed production lines in five files, plus 642 test
lines in two files; no generated files or documentation files. Creation
validation and SQL masking cover the supported job form; task lifecycle and TVF
execution handle cancellation, completion and cleanup; the tests cover those
paths and existing table jobs.
- **Before and after:** Wrapping a valid S3 INSERT in `CREATE JOB ... ON
SCHEDULE AT CURRENT_TIMESTAMP` was rejected. It now creates an asynchronous
one-time job. Recurring, delayed and STREAMING TVF jobs, and LOCAL/HDFS
destinations, are rejected during creation.
- **Configs and defaults:** No new configuration. Background tasks retain
the default session timeout. Existing table INSERT jobs keep their schedules,
label handling and execution path.
- **Impact on existing users:** Table jobs retain their behavior. Direct TVF
INSERTs also report timeout/backend failures correctly and participate in
cancellation and credential masking. External output keeps its existing
non-transactional semantics; cancellation or failure does not promise rollback
of partial files.
- **Behavior change:** The immediate S3 job form is supported; job creation
performs validation without destination writes/deletions, and canceled task
history is retained.
Example, using the same storage properties as a direct S3 INSERT:
```sql
CREATE JOB export_once ON SCHEDULE AT CURRENT_TIMESTAMP DO
INSERT INTO s3(
"file_path" = "s3://example-bucket/export/result_",
"format" = "csv",
"s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
"s3.region" = "us-east-1",
"s3.access_key" = "<access-key>",
"s3.secret_key" = "<secret-key>"
)
SELECT * FROM source_table;
```
### Testing
Locally on Linux x86_64 with ByteOpenJDK 17:
- Complete FE build through `build.sh --fe -j 4`, including packaging; UI
built from the current lockfile.
- FE Checkstyle: zero violations; `git diff --check` passed.
- Native `run-fe-ut.sh`: 123 tests across nine classes, with zero failures,
errors or skips. The 25 added tests cover validation without destination side
effects, schedule/target rejection, SQL masking after serialization, timeout
and backend errors, cancellation before/during planning and dispatch, resource
cleanup, terminal callbacks, persistence, and table INSERT compatibility. The
other 98 tests cover job status/configuration/authorization, existing INSERT
context creation, SQL encryption, `StmtExecutor` cancellation and table-stream
compatibility.
No external S3/MinIO end-to-end test was run in this checkout.
### Notes for Reviewers
The TVF command uses master's existing sticky cancellation callback, so
`StmtExecutor` needs no new special case. The current filesystem plugin path
and external DML audit coordinator are preserved. The added tests use Mockito,
including worker-local static/construction mocks, and restore the shared test
environment after each case.
### Release note
Support immediate, one-time jobs for `INSERT INTO s3(...) SELECT ...`, with
destination credential masking and cancellation/error handling. Other TVF job
schedules and LOCAL/HDFS job targets remain unsupported.
### Check List (For Author)
- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; follow-up website documentation should
add the S3-only, immediate-only JOB form and its restrictions, using the
example above.
- Pre-submission status: Ready.
### AI Disclosure
- Tool: OpenAI Codex (GPT-6).
- Generated: Conflict adaptations, migration of the existing tests to
Mockito, and this description.
- Human review: Not yet performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
