mixermt opened a new pull request, #7712:
URL: https://github.com/apache/datafusion-comet/pull/7712
## Which issue does this PR close?
Closes #7138.
## Rationale for this change
A data file can store a narrower type than its column: `INT` values in a
`BIGINT` column written before a type promotion, or `TINYINT`, `SMALLINT` or
`FLOAT` values in a file that `add_files`, `migrate`, `snapshot` or the Java
API added. At Comet's iceberg-rust pin, a range, `IN` or `!=` residual on such
a column drops matching rows:
- the page index skips pages whose `INT32`/`FLOAT` bounds it cannot compare
with the literal (fixed upstream by apache/iceberg-rust#3507, which landed
after the Arrow 60 bump, so the pin cannot take it yet);
- the row filter casts the literal down to the file's type, so an
out-of-range literal becomes NULL and a `double` rounds to `float`
(apache/iceberg-rust#3540, open).
The JVM cannot tell which files are narrower. Spark plans the scan without
column stats, and neither the schema history nor the name mapping records every
such file. #7156 and #7233 guard the cases that table metadata records, for the
whole table. This PR checks each file instead. It also covers promotions made
before Iceberg 0.12 and files appended without a name mapping, and files that
store the table's type keep their pruning.
## What changes are included in this PR?
- `iceberg_narrow_types.rs`: before a task reaches iceberg-rust's reader, if
its residual compares an `int`, `long` or `double` column with anything other
than `IS [NOT] NULL` or `=`, read the file's Parquet footer. The read uses a
512 KiB prefetch, skips page indexes, decrypts when needed, and happens once
per file. Each column is matched the way iceberg-rust matches it: by embedded
field ids, else through the name mapping, else treated as narrower. Every such
comparison on a column the file stores with another type is replaced by TRUE.
The residual has no `NOT`, so this only weakens it, and the filter above the
scan still applies the full predicate.
- New scan metric `narrow_type_splits`, the number of splits whose residual
was weakened.
- A note under "Filter pushdown" in the Iceberg user guide.
Cost: one extra footer read for each data file whose residual has such a
comparison. The guard can be removed once the pin includes #3507 and #3540.
## How are these changes tested?
- Rust unit tests in `iceberg_narrow_types.rs` cover:
- which operators count as unsafe;
- column matching: field ids, name mapping, mixed ids, no ids;
- weakening;
- end-to-end checks on real Parquet files: a narrower file is weakened, a
file with the table's types is left alone, a safe residual reads no footer, and
each footer is read once per file.
- `CometIcebergNativeSuite`, three new tests, each comparing range, `IN`,
`!=`, `NOT (=)` and equality predicates with Spark:
- a promoted table;
- an appended file whose field ids carry `INT`, `TINYINT`, `SMALLINT` and
`FLOAT`, with no name mapping;
- a name-mapped file without field ids.
They also assert that `narrow_type_splits` counts only the narrower file.
All three fail on `main`.
- `CometIcebergEncryptionSuite`: the footer check on an encrypted table,
before and after a promotion.
- Spark 4.1, JDK 17: `CometIcebergNativeSuite`,
`CometIcebergNativeScanSuite`, `CometFuzzIcebergSuite` and
`CometIcebergEncryptionSuite` ran 190 tests: 190 passed, 2 canceled by version
gates.
- Spark 3.5 and 4.0: the three new tests and `CometIcebergNativeScanSuite`
passed. The strict Scala warnings compile for Spark 3.5 passes.
- With the check disabled, the three new tests and the encryption test fail.
Not covered: equality deletes on such a file. iceberg-rust builds their
predicate inside the reader, so it can still drop rows there until the upstream
fix lands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]