zhangfengcdt opened a new pull request, #6754: URL: https://github.com/apache/datafusion-comet/pull/6754
## Which issue does this PR close? Closes #6707 ## Rationale for this change The native Iceberg scan is on by default. Under it, `input_file_name()`, `input_file_block_start()` and `input_file_block_length()` return `""`, `-1` and `-1` for every row, and the query still succeeds. These functions read `InputFileBlockHolder`, which Iceberg's Spark reader sets as it opens each data file. The native scan never sets it. `transformV1Scan` already falls back from the native Parquet scan when the plan uses these functions (#3312). `transformV2Scan` had no such check and was not given the plan, so it could not make one. ## What changes are included in this PR? - `transformV2Scan` now receives the plan, as `transformV1Scan` does. After the Iceberg metadata-table check, it falls back to Spark's reader when any node in the plan uses one of the three functions. The check sits where all of Comet's built-in V2 scans pass, so it also covers the native CSV V2 scan, which is off by default. - The check is inlined, matching the V1 predicate. #6703 adds `CometScanRule.readsInputFileBlock` for the same predicate. Whichever of the two merges second can switch this check to that helper. It is a candidate for `backport-1.1`. ## How are these changes tested? A new test in `CometIcebergNativeSuite` writes three data files and runs the issue's query, with and without a filter above the scan. Both must match Spark and fall back from the native Iceberg scan. A query without the functions must still use the native scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
