[
https://issues.apache.org/jira/browse/SPARK-31798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-31798:
-------------------------------
Description:
Part of the overall sub-push for shuffle metadata management as proposed in
[this
document|https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit].
The shuffle plugin system needs a means for storing custom metadata for shuffle
readers to eventually use. This task tracks the API changes required for the
map output writer to return some such metadata back to the driver.
was:
Part of the overall sub-push for shuffle metadata management as proposed in
[this
document|[https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit]]
The shuffle plugin system needs a means for storing custom metadata for shuffle
readers to eventually use. This task tracks the API changes required for the
map output writer to return some such metadata back to the driver.
> Return map output metadata from shuffle writers
> -----------------------------------------------
>
> Key: SPARK-31798
> URL: https://issues.apache.org/jira/browse/SPARK-31798
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
> Affects Versions: 3.0.0
> Reporter: Matt Cheah
> Priority: Major
>
> Part of the overall sub-push for shuffle metadata management as proposed in
> [this
> document|https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit].
> The shuffle plugin system needs a means for storing custom metadata for
> shuffle readers to eventually use. This task tracks the API changes required
> for the map output writer to return some such metadata back to the driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
