morningman opened a new pull request, #69691:
URL: https://github.com/apache/doris/pull/69691
### What problem does this PR solve?
Issue Number: None
Related PR: #66538 (branch-4.1), #68267
Problem Summary:
**In short.** This brings the V1-reader part of branch-4.1 #66538 (Iceberg
V3 default values) to master: Iceberg equality deletes keep working after the
type of their key column is promoted (INT to BIGINT, FLOAT to DOUBLE, a wider
DECIMAL precision). It does not copy branch-4.1's approach, which compares a
key in the delete file's older type; it compares in the scan schema's type, as
#68267 already does in the V2 reader.
**Background.** Iceberg allows widening a column's type, and files keep the
type they were written with: an equality delete written while `id` was `int`
stores its key as INT32, and so does a data file written then. The V1 reader
(`enable_file_scanner_v2 = false`) loads each delete file into a delete block
and compares it with the data rows, which the Parquet and ORC readers already
return in the table's current type.
**The problem, and what it cost.** The V1 reader compared every key in the
type of the delete file it came from. After a promotion, any of these fails:
| Case (V1 reader, `id` or `payload.existing` promoted from INT to BIGINT) |
master |
|---|---|
| `SELECT id`, delete written before the promotion | Query fails: `Bad cast
from type ColumnVector<BIGINT> to ColumnVector<INT>` |
| `SELECT id`, one delete written before the promotion and one after | Query
fails: the INT and BIGINT keys are merged into one delete block (debug builds
abort on the block's type check) |
| `SELECT v` (key not projected), same two deletes | Same |
| `SELECT id`, delete on `payload.existing` written after the promotion,
data file written before it | The INT values of the old file are copied into a
BIGINT column: debug builds abort with `Bad cast`, release builds read the
wrong bytes and keep or delete the wrong rows |
branch-4.1 / branch-4.2 handle these by casting the data key down to the
delete file's type. That narrowing is what #68267 fixed in the V2 reader: a
data file written after the promotion can hold 4294967296, which becomes NULL
as INT and is then deleted by the NULL key of a delete file written before the
promotion (equality deletes use null-safe equality); with `enable_strict_cast`
the whole scan fails instead.
**How this PR fixes it.**
- The type a key is compared in is the field's type in the scan schema
(`equality_delete_key_type`, the same rule as V2's
`restore_current_primitive_type`). A plan without a scan schema keeps the
delete file's type, as before, and STRING / VARBINARY keys keep theirs too,
since they match by bytes.
- Delete keys, and nested keys extracted from the physical root of an older
data file, are widened to that type with `converter::ColumnTypeConverter`, the
converter the V1 Parquet and ORC readers already apply to the data column of a
promoted field, so both sides hold the same value
(`widen_equality_delete_key`). `NestedEqualityDeleteColumn` records the leaf's
source type for that.
- Delete blocks and hidden key carriers are built in the comparison type, so
deletes written before and after a promotion share one block.
**Results.**
| Case (V1 reader) | master | This PR |
|---|---|---|
| `SELECT id`, INT delete `id = 2`, data 1, 2, 3 | `Bad cast` | 1, 3 |
| `SELECT id`, INT delete `id = 2` + BIGINT delete `id = 3` | aborts (debug)
/ fails | 1 |
| `SELECT v`, same two deletes | aborts (debug) / fails | 10 |
| `SELECT id`, BIGINT delete `payload.existing = 20` on an old INT file |
`Bad cast` (debug) / wrong rows (release) | 1, 3 |
| `SELECT id`, data `NULL, 0, 1, 4294967296`, INT delete holding NULL | `Bad
cast` (branch-4.x: 0, 1) | 0, 1, 4294967296 |
### Release note
Fix Iceberg equality deletes on the V1 file scanner (`enable_file_scanner_v2
= false`) failing, or keeping and deleting the wrong rows, after the type of an
equality-key column was promoted.
### Check List (For Author)
- Test: Unit Test
- 9 new `IcebergReaderTest` cases, Parquet and ORC: a projected key with
a delete written before the promotion; deletes written before and after it with
the key projected and hidden; a nested key deleted after the promotion against
an older data file; an out-of-range value of a promoted key against a NULL
delete key. All 9 fail on master (`Bad cast` or the delete block's type check)
and pass with this PR.
- `IcebergReaderTest`, `EqualityDeleteTest`,
`IcebergReaderCreateColumnIdsTest`, `ParquetReadLinesTest`,
`IcebergPositionDeleteSys*`, `VIcebergDeleteSinkTest`,
`IcebergTableReaderTest`, `IcebergV2ReaderTest` and the table schema change
helper tests pass.
- No regression test: Spark SQL cannot write equality deletes, and no
regression fixture has equality deletes on a promoted column.
- Behavior changed: Yes. V1 equality deletes on a promoted key column now
apply instead of failing the query or matching the wrong rows.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
