Vivek1106-04 commented on PR #57528: URL: https://github.com/apache/spark/pull/57528#issuecomment-5306876605
@uros-b added an entry to `docs/core-migration-guide.md`. Since `branch-4.3` is already cut, it opens a new "Upgrading from Core 4.3 to 4.4" section: > Since Spark 4.4, Spark accurately records skewed shuffle block sizes in `HighlyCompressedMapStatus`, so that adaptive query execution can still detect skewed partitions in shuffles with more than 2000 partitions. The default of `spark.shuffle.accurateBlockSkewedFactor` changed from `-1.0` to `5.0`, matching `spark.sql.adaptive.skewJoin.skewedPartitionFactor`. This can change query plans, since such skewed partitions are now split by skew join optimization, and it retains up to `spark.shuffle.maxAccurateSkewedBlockNumber` (100 by default) block sizes per map task on the driver. To restore the legacy behavior, you can set `spark.shuffle.accurateBlockSkewedFactor` to `-1.0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
