morningman opened a new pull request, #69536:
URL: https://github.com/apache/doris/pull/69536
### What problem does this PR solve?
Issue Number: None
Related PR: #66538, #69394
Problem Summary:
**In short.** This ports a fix from branch-4.1 #66538 (Iceberg V3 default
values). When a dropped column is still referenced by a live equality delete,
the Iceberg connector now looks up the column's definition along the scan
snapshot's parent chain first, and only then in the whole schema list. Master
searched only the part of the schema list before the scan schema. That breaks
as soon as Iceberg reuses a schema ID, and every query on such a table failed.
**Background.** An equality delete names its key columns by field ID. If a
key column has since been dropped, the scan schema no longer has it, so the
connector adds the column's historical definition to the schema dictionary it
sends to BE (`IcebergScanPlanProvider.schemaForPotentialEqualityDeletes`). BE
needs that field ID, type and initial default to read the key from data files.
Master (#65851) took the definition from the metadata schema list, searching
backwards from the scan schema's position, on the assumption that the list is
in chronological order.
**The problem, and what it cost.** Schema IDs are not a sequence. When a
schema change produces a schema equal to an existing one, Iceberg reuses that
schema's ID and does not append to the list. Dropping a column that was added
after the table was created is the common case: the schema returns to the
initial one, so the current schema is the first entry of the list, and the
schema that still defines the key comes after it. The backward search never
reaches that schema, and the scan fails.
Reproduce: create a table with `id`; add column `k`; write data and an
equality delete keyed on `k`; drop `k`; query the table.
| | Master today | After this PR |
|---|---|---|
| Query after dropping an equality-delete key, when the schema returns to an
earlier one | fails: `Iceberg equality-delete fields are absent from schema
history: [2]` | succeeds |
| Same, after the snapshots that recorded the key have expired | fails with
the same error | succeeds (the key comes from the metadata schema list) |
| Time travel to a snapshot whose lineage defines the key, while a later
schema renames it | earlier definition | earlier definition (unchanged) |
**How this PR fixes it.** It uses the same order as
`IcebergScanNode.getSchemaFieldsForScan` on branch-4.1 and branch-4.2. First,
walk the scan snapshot's parent chain and take each missing key from the schema
that snapshot was written with. Then search the whole metadata schema list,
newest first, for keys that only schema-only changes or expired ancestors can
explain. Unlike branch-4.x, the walk indexes each schema at most once and stops
as soon as every key is resolved, so a long lineage that shares one schema
costs only map lookups. When no key is missing, nothing changes and no snapshot
is read, as before.
When a dropped key has several historical definitions, the one recorded on
the scan's lineage now wins, as on branch-4.x. The field ID and the initial
default are the same either way. Only a name or optionality changed by a
schema-only commit after the last write can differ. The existing test
`equalityCarrierKeepsLatestDefinitionOfDroppedEqualityKey` (from #69394)
renamed and then dropped the key with no write in between. It now writes a
snapshot after each schema change, as the branch-4.1 test it was ported from
does, and still expects `k2`.
**Results.** This PR ports four tests from #66538 into
`IcebergScanPlanProviderTest`:
- `equalityCarrierResolvesDroppedKeyWhenSchemaIdIsReused` uses a real
in-memory table and also checks the scan dictionary that
`getScanNodeProperties` builds.
- `equalityCarrierResolvesReusedSchemaIdAfterParentSnapshotsExpire` is the
same, with the snapshots that recorded the key expired.
- `equalityCarrierPrefersScanLineageOverLaterRenameForTimeTravel` and
`equalityCarrierKeepsDefaultWhenTimeTravelParentExpiredBeforeLaterRename` use
table doubles with schema histories that the SDK cannot build.
With master's search, the two reused-ID tests fail with the error above.
Without the lineage walk, the time-travel test gets `k2` instead of `k`. All
1693 tests of fe-connector-iceberg pass.
Original author: @hubgeter
### Release note
Fix: queries on an Iceberg table failed with "equality-delete fields are
absent from schema history" after a column used as an equality-delete key was
dropped and the table schema returned to an earlier one.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
Such queries no longer fail. When a dropped equality-delete key has
several historical definitions, the one recorded on the scan snapshot's lineage
wins, as on branch-4.x.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
