andygrove opened a new issue, #3312: URL: https://github.com/apache/datafusion-comet/issues/3312
## Summary 3 Spark SQL tests fail because `input_file_name()` returns an empty string when `native_datafusion` scan is used. ## Failing Tests - `UDFSuite`: "SPARK-8005 input_file_name" - `ColumnExpressionSuite`: "input_file_name, input_file_block_start, input_file_block_length - FileScanRDD" - `ExtractPythonUDFsSuite`: "Python UDF should not break column pruning/filter pushdown -- Parquet V1" ## Root Cause `native_datafusion` scan doesn't populate the `input_file_name()` / `input_file_block_start()` / `input_file_block_length()` metadata that Spark's `FileSourceScanExec` provides. This is a functional gap. ## Possible Fix In `CometScanRule.nativeDataFusionScan()`, detect when the query references `InputFileName`, `InputFileBlockStart`, or `InputFileBlockLength` expressions and fall back to `native_iceberg_compat`. ## Related Discovered in CI for #3307 (enable native_datafusion in auto scan mode). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
