Tim Lee created SPARK-49741:
-------------------------------
Summary: Add spark.shuffle.accurateBlockSkewedFactor to OSS Apache
Spark configuration docs page
Key: SPARK-49741
URL: https://issues.apache.org/jira/browse/SPARK-49741
Project: Spark
Issue Type: Task
Components: Documentation
Affects Versions: 3.5.0, 3.4.0, 3.3.0
Reporter: Tim Lee
spark.shuffle.accurateBlockSkewedFactor was added in Spark 3.3.0 in
https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle
configuration to prevent issues where HighlyCompressedMapStatus underestimates
the shuffle block sizes when the size distribution is skewed. Currently this
config is not discoverable because it's not on the Spark config docs page and
we should add it there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
