hubgeter opened a new pull request, #68267:
URL: https://github.com/apache/doris/pull/68267
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Iceberg equality deletes are compared in the physical type of the delete
file.
When a column is legally promoted (for example INT to LONG), the data file
can
hold values outside the old type, but the data key was cast into that old
type.
An out-of-range value becomes NULL and then matches a NULL equality-delete
key
through NULL-safe equality, so a valid row is silently deleted. With
`enable_strict_cast` the same cast makes the whole scan fail with
`Value ... out of range for type int`.
Resolve the comparison type from the current snapshot schema instead of the
delete file, and promote historical delete values into that domain. With both
sides in the wider type the cast is always lossless for an Iceberg-legal type
promotion, so a non-NULL key can no longer turn into NULL. Include the
current
schema id in the delete-file cache key because the loaded block now depends
on
the comparison type.
Covered by a new BE unit test: a data file holding LONG `NULL, 0, 1,
4294967296`
plus a NULL equality-delete key written under the old INT schema. Before the
fix
the scan returned `{0, 1}`; after the fix it returns `{0, 1, 4294967296}`.
### Release note
Fix Iceberg equality deletes being evaluated in the delete file's historical
type, which could silently drop valid rows after a column type promotion.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
