andygrove opened a new issue, #7039: URL: https://github.com/apache/datafusion-comet/issues/7039
### What is the problem the feature request solves? When the native Iceberg scan needs no data column from a file, it reads every column of that file instead of none. That happens for `count(*)` and `SELECT 1` on tables whose files have deletes (Iceberg answers a plain `count(*)` from metadata), and for a query whose columns were all added after the file was written. Comet sends an empty `project_field_ids` list for an empty projection (`CometIcebergNativeScan.scala` around line 1188). iceberg-rust turns an empty list into `ProjectionMask::all()`, and does the same when none of the requested field ids exists in the file (`crates/iceberg/src/arrow/reader/projection.rs` around lines 122-124 and 231-235 at the current pin). Iceberg Java reads no data columns in these cases. Measured on `main` at 66c63bd307 (Spark 4.1.3, Iceberg 1.11) over a merge-on-read table of 20,000 rows with a deletion. The numbers are the native scan's `bytes_scanned`: | Query | `bytes_scanned` | | --- | --- | | `SELECT count(id)` | 574,916 | | `SELECT count(*)` | 1,563,258 | | `SELECT 1 ... LIMIT 5` | 1,563,258 | | `SELECT id` after `ADD COLUMN late` and one more insert | 576,293 | | `SELECT late` | 1,564,635 | A related case, by code reading only: iceberg-rust decodes every column of a delete file (`BasicDeleteFileLoader::parquet_to_batch_stream` in `crates/iceberg/src/arrow/delete_file_loader.rs` sets no projection, at the current pin and on iceberg-rust main). Equality-delete files that carry whole rows, as Flink's non-upsert CDC writer produces, or position-delete files with a `row` column are decoded in full, where Iceberg Java projects the key or `file_path` and `pos` columns. ### Describe the potential solution For the empty projection, Comet could ask for a metadata column such as `_pos` instead of an empty list. iceberg-rust reads no data column for that, and the zero-column path in `IcebergStreamWrapper` keeps only the row count. That was not tried. iceberg-rust could also treat an empty projection, and one with no matching column, as "read nothing". Projecting delete files to the columns the delete needs belongs in iceberg-rust. ### Additional context Found while auditing nested types in the native scans, although none of this is specific to nested types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
