JNSimba opened a new pull request, #68007:
URL: https://github.com/apache/doris/pull/68007
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: S3 streaming insert jobs currently keep polling for new
files after consuming all files matched by the path. This change adds
`s3.ingestion_mode=ONCE`, which imports matching files in lexical batches and
marks the job as `FINISHED` after the final batch commits successfully.
Recovered jobs probe from their committed offset and finish when no files
remain. ONCE jobs reject user-specified offsets, including offset changes
through ALTER JOB.
### Release note
S3 streaming insert jobs support `s3.ingestion_mode=ONCE` for one-time,
batched ingestion.
### Check List (For Author)
- Test
- [x] Unit Test
- 19 tests passed: S3 offset batching and recovery, scheduler
completion, task commit handling, and property validation.
- [ ] Regression test
- [ ] Manual test
- [ ] No need to test or manual test.
- Behavior changed:
- [x] Yes. S3 streaming jobs configured with ONCE finish after all
discovered lexical batches are committed.
- [ ] No.
- Does this need documentation?
- [x] Yes. The new S3 ingestion mode needs to be added to the Streaming
Job documentation.
- [ ] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
