[
https://issues.apache.org/jira/browse/SPARK-49741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Lee updated SPARK-49741:
----------------------------
Description: spark.shuffle.accurateBlockSkewedFactor was added in Spark
3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful
shuffle configuration to prevent issues where HighlyCompressedMapStatus wrongly
estimates the shuffle block sizes when the size distribution is skewed.
Currently this config is not discoverable because it's not on the Spark config
docs page and we should add it there. (was:
spark.shuffle.accurateBlockSkewedFactor was added in Spark 3.3.0 in
https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle
configuration to prevent issues where HighlyCompressedMapStatus underestimates
the shuffle block sizes when the size distribution is skewed. Currently this
config is not discoverable because it's not on the Spark config docs page and
we should add it there.)
> Add spark.shuffle.accurateBlockSkewedFactor to OSS Apache Spark configuration
> docs page
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-49741
> URL: https://issues.apache.org/jira/browse/SPARK-49741
> Project: Spark
> Issue Type: Task
> Components: Documentation
> Affects Versions: 3.3.0, 3.4.0, 3.5.0
> Reporter: Tim Lee
> Priority: Major
>
> spark.shuffle.accurateBlockSkewedFactor was added in Spark 3.3.0 in
> https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle
> configuration to prevent issues where HighlyCompressedMapStatus wrongly
> estimates the shuffle block sizes when the size distribution is skewed.
> Currently this config is not discoverable because it's not on the Spark
> config docs page and we should add it there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
