[
https://issues.apache.org/jira/browse/SPARK-57993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57993:
-----------------------------------
Labels: pull-request-available (was: )
> Support advisory partition size in rebalance hint
> -------------------------------------------------
>
> Key: SPARK-57993
> URL: https://issues.apache.org/jira/browse/SPARK-57993
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Zhen Wang
> Priority: Major
> Labels: pull-request-available
>
> Spark SQL users often insert a *REBALANCE* hint before writing query results
> to control small files and produce output partitions with a reasonable target
> size.
> Today, the target size used by rebalance is controlled by the session-level
> *spark.sql.adaptive.advisoryPartitionSizeInBytes* configuration. This is too
> broad for write-side file size control: increasing the advisory partition
> size for the final write stage can also affect earlier AQE stages, including
> shuffle coalescing and read/computation parallelism before the final output
> stage.
> This proposes adding a *REBALANCE_BY_SIZE* hint that works like *REBALANCE*
> and accepts an optional advisory partition size in bytes as the first
> parameter, followed by optional partition columns. Users can insert this hint
> before writing and set the target partition size only for the rebalance
> shuffle, so the final output file size can be controlled without changing the
> advisory size used by preceding stages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
