rekbun commented on code in PR #59522: URL: https://github.com/apache/spark/pull/59522#discussion_r4446990151
########## docs/sql-migration-guide.md: ########## @@ -106,6 +106,8 @@ license: | ## Upgrading from Spark SQL 4.1 to 4.2 +- Since Spark 4.2, V2 scans no longer unconditionally group input splits sharing a reported partition key into one task (SPARK-55092). Spark still groups splits when needed to satisfy a downstream distribution, such as for a storage-partitioned join. Other scans can execute one task per split, increasing overhead for connectors with many small splits per key. Before 4.2, grouping multiple splits also discarded their reported ordering. Spark 4.4 introduces the default-off `spark.sql.sources.v2.scanPartitionPacking.enabled` option to pack eligible batch V2 scan inputs by estimated size, with an explicit count fallback when sizes are unavailable. This policy is independent of partition keys and does not restore unconditional per-key grouping. See [DataSource V2 Scan Task Sizing](sql-performance-tuning.html#datasource-v2-scan-task-sizing) for its scope and tradeoffs. Review Comment: Done: removed here and moved to #59970 (SPARK-60763). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
