JNSimba opened a new pull request, #69363:
URL: https://github.com/apache/doris/pull/69363
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
S3 streaming LIST ingestion drops the modification time when converting
listed files to scan metadata. After a failed object is overwritten, a resumed
job can reuse cached bytes for the old object because its cache identity has
not changed. EVENT ingestion also uses the size from the notification, which
may describe an older object and truncate the scan of its replacement.
Preserve the listed modification time and fetch the current object size and
modification time with HEAD when accepting an EVENT record. Skip an explicitly
missing object; propagate other HEAD failures so uncommitted notifications
remain available for retry. Perform HEAD outside the provider lock and discard
late batches after close. Objects must still remain unchanged while a batch is
being ingested.
Outbound requests / SSRF surface: EVENT processing adds HEAD requests to the
S3 endpoint configured in the job's TVF, using the existing object storage
client and credentials. Streaming job creation is accessible to users with LOAD
privilege on the destination table; SUPER privilege is not required.
### Release note
Fix stale S3 streaming source metadata when retrying ingestion after an
object is overwritten.
### Check List (For Author)
- Test:
- [x] Unit Test
- [x] Regression test
- [x] Manual test
- Targeted FE UTs after resolving the rebase:
`S3SourceOffsetProviderTest` (7) and `S3EventSourceOffsetProviderTest` (47), 54
tests passed, 0 failures/errors/skipped, using `run-fe-ut.sh --run`.
- Temporary ONE_TIME regression passed five consecutive runs on a
shared-nothing 1 FE / 1 BE cluster, with file cache enabled. Each run verified
old bad-object cache blocks, replacement object metadata, successful RESUME,
all 30 expected rows, and coexistence of old/new cache generations. The
temporary suite is not included in this PR.
- FE build and 51 targeted UTs passed before rebasing. After rebasing,
the four updated S3 source/test files are tested over the existing developer
build (baseline `1a1f93842d8`); the complete reactor with the latest master's
dependency versions has not been rebuilt.
- Checkstyle on the rebased PR branch: 0 violations; `git diff --check`
passed.
- Shared-storage deployment and EVENT end-to-end ingestion were not
tested.
- Behavior changed:
- [x] Yes. EVENT batches use current object metadata rather than
notification metadata; missing objects are skipped and other HEAD failures
remain retryable.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
