andygrove opened a new pull request, #6875: URL: https://github.com/apache/datafusion-comet/pull/6875
## Which issue does this PR close? Closes #6573. Backport of #6703 to `branch-1.1`. ## Rationale for this change Spark's input-file expressions read thread-local state maintained by the source reader. Wrapping a Spark scan in a row-to-Arrow conversion reads ahead and can make rows report the wrong file name, block offsets, or unset defaults. The fix should be available in the 1.1 release line. ## What changes are included in this PR? Cherry-picks #6703 with `-x`, sharing the input-file-expression detector between the native scan and execution rules and refusing Spark-to-Arrow conversion when those expressions are present. Regression tests cover V1/V2 Parquet, CSV, JSON, and RDD scans. The 1.1 branch predates typed-Dataset output conversion and the per-source RDD conversion config. The backport therefore omits the inapplicable typed-Dataset path/test and uses the branch's existing `spark.comet.sparkToColumnar.enabled` setting for the RDD regression. Documentation is narrowed accordingly. ## How are these changes tested? Ran: ```text ./mvnw test -Dtest=none -Dsuites='org.apache.comet.exec.CometExecSuite input_file_name above a converted' -Dscalastyle.skip=true ``` Result: 1 suite completed, 2 focused tests passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
