mixermt opened a new pull request, #7712:
URL: https://github.com/apache/datafusion-comet/pull/7712

   ## Which issue does this PR close?
   
   Closes #7138.
   
   ## Rationale for this change
   
   A data file can store a narrower type than its column: `INT` values in a 
`BIGINT` column written before a type promotion, or `TINYINT`, `SMALLINT` or 
`FLOAT` values in a file that `add_files`, `migrate`, `snapshot` or the Java 
API added. At Comet's iceberg-rust pin, a range, `IN` or `!=` residual on such 
a column drops matching rows:
   
   - the page index skips pages whose `INT32`/`FLOAT` bounds it cannot compare 
with the literal (fixed upstream by apache/iceberg-rust#3507, which landed 
after the Arrow 60 bump, so the pin cannot take it yet);
   - the row filter casts the literal down to the file's type, so an 
out-of-range literal becomes NULL and a `double` rounds to `float` 
(apache/iceberg-rust#3540, open).
   
   The JVM cannot tell which files are narrower. Spark plans the scan without 
column stats, and neither the schema history nor the name mapping records every 
such file. #7156 and #7233 guard the cases that table metadata records, for the 
whole table. This PR checks each file instead. It also covers promotions made 
before Iceberg 0.12 and files appended without a name mapping, and files that 
store the table's type keep their pruning.
   
   ## What changes are included in this PR?
   
   - `iceberg_narrow_types.rs`: before a task reaches iceberg-rust's reader, if 
its residual compares an `int`, `long` or `double` column with anything other 
than `IS [NOT] NULL` or `=`, read the file's Parquet footer. The read uses a 
512 KiB prefetch, skips page indexes, decrypts when needed, and happens once 
per file. Each column is matched the way iceberg-rust matches it: by embedded 
field ids, else through the name mapping, else treated as narrower. Every such 
comparison on a column the file stores with another type is replaced by TRUE. 
The residual has no `NOT`, so this only weakens it, and the filter above the 
scan still applies the full predicate.
   - New scan metric `narrow_type_splits`, the number of splits whose residual 
was weakened.
   - A note under "Filter pushdown" in the Iceberg user guide.
   
   Cost: one extra footer read for each data file whose residual has such a 
comparison. The guard can be removed once the pin includes #3507 and #3540.
   
   ## How are these changes tested?
   
   - Rust unit tests in `iceberg_narrow_types.rs` cover:
     - which operators count as unsafe;
     - column matching: field ids, name mapping, mixed ids, no ids;
     - weakening;
     - end-to-end checks on real Parquet files: a narrower file is weakened, a 
file with the table's types is left alone, a safe residual reads no footer, and 
each footer is read once per file.
   - `CometIcebergNativeSuite`, three new tests, each comparing range, `IN`, 
`!=`, `NOT (=)` and equality predicates with Spark:
     - a promoted table;
     - an appended file whose field ids carry `INT`, `TINYINT`, `SMALLINT` and 
`FLOAT`, with no name mapping;
     - a name-mapped file without field ids.
   
     They also assert that `narrow_type_splits` counts only the narrower file. 
All three fail on `main`.
   - `CometIcebergEncryptionSuite`: the footer check on an encrypted table, 
before and after a promotion.
   - Spark 4.1, JDK 17: `CometIcebergNativeSuite`, 
`CometIcebergNativeScanSuite`, `CometFuzzIcebergSuite` and 
`CometIcebergEncryptionSuite` ran 190 tests: 190 passed, 2 canceled by version 
gates.
   - Spark 3.5 and 4.0: the three new tests and `CometIcebergNativeScanSuite` 
passed. The strict Scala warnings compile for Spark 3.5 passes.
   - With the check disabled, the three new tests and the encryption test fail.
   
   Not covered: equality deletes on such a file. iceberg-rust builds their 
predicate inside the reader, so it can still drop rows there until the upstream 
fix lands.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to