rich7420 commented on code in PR #6847:
URL: https://github.com/apache/datafusion-comet/pull/6847#discussion_r4349337145
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -1009,6 +1082,18 @@ object CometIcebergNativeScan extends
CometOperatorSerde[CometBatchScanExec] wit
// schema so a filter column projected out of the scan output is still
recognized.
val pageIndexUnsupportedColumns =
IcebergReflection.pageIndexUnsupportedColumns(metadata.tableSchema)
+ // Columns whose type changed over the table's history; residual
predicates comparing them
+ // with a literal are dropped, except equality. The scan schema also
covers a column that
+ // VERSION AS OF reads after it was dropped from the table.
+ val promotedColumns = Seq(metadata.tableSchema, metadata.scanSchema)
+ .flatMap(IcebergReflection.promotedColumns(metadata.table, _))
+ .toSet
+ // add_files, migrate and snapshot set a name mapping and import files
without checking their
+ // types against the schema. Such files commonly hold narrower physical
types, such as Hive
+ // TINYINT and SMALLINT stored as INT32 with INT_8 or INT_16 annotations,
which iceberg-rust
+ // compares incorrectly. So partial pushdown is not extended to these
tables, and they push a
+ // residual only when it converts whole.
+ val partialPushdown = metadata.nameMapping.isEmpty
Review Comment:
Thanks, confirmed. Before this PR the array null check kept the whole
residual out of the scan, so this was a regression.
Fixed in b5fc1be16. Residuals that convert in full are unchanged from #7156.
Otherwise, on tables without a name mapping, a partial pass treats every
`BIGINT` and `DOUBLE` column as promoted and keeps only their null checks and
`=` outside `NOT`. `DECIMAL` predicates are still not pushed.
The regression test `partial residuals do not compare columns promoted
before Iceberg 0.12` reads an `INT`/`FLOAT` file in a `BIGINT`/`DOUBLE` table
with one schema and no name mapping. Without the fix, `l IS NOT NULL AND id >
0` returns 0 rows instead of 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
