Gabriel39 opened a new pull request, #66348:
URL: https://github.com/apache/doris/pull/66348
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Iceberg writes had several correctness and resource-safety gaps across the
insert lifecycle:
- failures before execution could bypass the normal rollback and cleanup
path;
- failures after a durable external commit could incorrectly report the
write as failed;
- overwrite operations did not validate conflicts against their begin-time
snapshot;
- multi-partition sorted writes accounted for only the most recent
partition's revocable memory;
- an underlying partition-writer close error could be hidden by deferred
cleanup;
- large commit metadata could exceed the Thrift report limit after remote
files were written;
- failed writer initialization and multipart uploads could leave orphan
files;
- partition dispatch allocated full-block filters for every partition.
This change unifies lifecycle cleanup, makes post-commit refresh/listener
work best-effort, adds overwrite OCC validation, publishes all active partition
writers for memory accounting, propagates close failures, bounds commit
metadata before reporting, adds explicit writer abort and orphan cleanup
support, and dispatches rows with compact permutations.
### Release note
Harden Iceberg write correctness, memory accounting, and orphan-file cleanup.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. External commits are not reported as failed because of
post-commit cache/listener errors; concurrent overwrites now fail conflict
validation; incomplete files and multipart uploads are cleaned up.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
