andygrove opened a new pull request, #7018: URL: https://github.com/apache/datafusion-comet/pull/7018
## Which issue does this PR close? Closes #6707 on `branch-1.1`. #6754 already closed it on `main`. ## Rationale for this change This backports #6754 to `branch-1.1`. The native Iceberg scan is on by default (`spark.comet.scan.icebergNative.enabled=true`). It does not set `InputFileBlockHolder`, which Iceberg's Spark reader sets as it opens each data file. So `input_file_name()`, `input_file_block_start()` and `input_file_block_length()` over an Iceberg table return `""`, `-1` and `-1` for every row, with no error. The native CSV V2 scan has the same problem, but it is off by default. `transformV1Scan` already falls back from the native Parquet scan when the plan uses these expressions (#3312), and `transformV2Scan` had no such check. It is a silent wrong-result bug (`priority:critical`, `correctness`) that default settings reach. It is not a 1.1.0 regression, because the native Iceberg scan has had the same gap since it became the default. I am proposing it because of the silent wrong results, in the same way as #6270 and #6700. ## What changes are included in this PR? One cherry-pick with `-x`. See #6754 for the details. `transformV2Scan` now takes the whole plan and declines the scan, with a fallback reason, when the plan reads `input_file_name`, `input_file_block_start` or `input_file_block_length`. The Iceberg and CSV V2 scans then stay in Spark's reader. The `CometScanRule`, `iceberg.md` and `datasources.md` changes are identical to `main`. One adaptation, recorded in the commit message: `CometCsvNativeReadSuite` on this branch lacks the timestamp and TIME tests that #6754 adds its CSV test next to, so the cherry-pick conflicted there. The resolution keeps only the new `input_file_name` test and adds the `CometCsvNativeScanExec` import it needs. ## How are these changes tested? Run locally on this branch with JDK 17, on the default Spark 4.1 profile: - `CometCsvNativeReadSuite` (4 tests, including the new one) and the `iceberg/input_file_name.sql` fixture in `CometSqlFileTestSuite` pass. Scalastyle and spotless ran in the same build. - On Spark 3.5 and Scala 2.12, the sources and tests compile. - With the `CometScanRule` change taken out, both new tests fail. The other three pass. By CI's path routing, which I checked with `dev/ci/compute-changes.py`, this PR runs the all-profiles Linux build, macOS, the Spark SQL jobs for 3.5, 4.0 and 4.1, and the four Iceberg versions, so no `run-*` label is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
