goutamadwant opened a new pull request, #57809: URL: https://github.com/apache/spark/pull/57809
### What changes were proposed in this pull request? Run `SchemaPruning` after `PushVariantIntoScan` in the early scan pushdown batch. This removes variant reconstruction projections that become unnecessary during variant pushdown and allows the `Once` batch to reach the same plan on its first application. Add a Parquet V1 regression test covering a query that reads `_metadata.file_path` while a VARIANT column is referenced below a nondeterministic filter. Closes #57659. ### Why are the changes needed? `PushVariantIntoScan` can make a variant reconstruction projection unnecessary after the earlier schema-pruning passes have completed. Reapplying the optimizer batch then removes that projection, which violates the idempotence requirement for a `Once` batch and causes the reported query to fail during optimization. Running `SchemaPruning` once after variant pushdown makes the first application reach the stable plan. ### Does this PR introduce _any_ user-facing change? Yes. Queries that select file metadata while pruning a VARIANT column below a nondeterministic filter no longer fail during optimization with a `Once` batch idempotence error. ### How was this patch tested? The new regression test was confirmed to fail before the optimizer change and pass afterward. The following tests and checks passed: - `./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite -- -z "SPARK-57659"'` - `./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanVectorizedSuite'` (349 tests) - `./build/sbt 'sql/scalastyle' 'sql/Test/scalastyle'` - `git diff --check` The full `./dev/run-tests` suite was not run locally. ### Was this patch authored or co-authored using generative AI tooling? AI was used to review the code and understand the existing codebase. This contribution is my original work, and I license it under the project's open source license. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
