spark-test commented on PR #59732:
URL: https://github.com/apache/spark/pull/59732#issuecomment-6364008158
Confirmed: `spark.sql.sources.useV1SourceList` defaults to
`avro,csv,json,kafka,orc,parquet,text`, so the built-in file sources use the V1
path by default, and the V1 path already evaluates these Python UDF predicates
after the scan. Only reads that route a format to Data Source V2 are affected.
If this lands for 4.4, the migration-guide entry should move from the
4.4-to-5.0 section to 'Upgrading from Spark SQL 4.3 to 4.4' and say 'Since
Spark 4.4'; see the review comment on `docs/sql-migration-guide.md`.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6355041388","thread_id":"conversation:6355041388","verdict_sha256":"40900ffa5ff23f52ef4554157593bfb7001f2c2f321334a60cec88b7d8c4cbf5"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
