[
https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422627#comment-17422627
]
Guowei Ma commented on FLINK-13247:
-----------------------------------
Hi [~dangshazi]
I am glad you are interested in the this topic. Currently there is no public
doc yet.
But we are plan to open source our remote shuffle service in mid-to-late
October.
> Implement external shuffle service for YARN
> -------------------------------------------
>
> Key: FLINK-13247
> URL: https://issues.apache.org/jira/browse/FLINK-13247
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: MalcolmSanders
> Priority: Minor
> Labels: auto-deprioritized-major
>
> Flink batch job users could achieve better cluster utilization and job
> throughput throught external shuffle service because the producers of
> intermedia result partitions can be released once intermedia result
> partitions have been persisted on disks. In
> [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang]
> has introduced pluggable shuffle manager architecture which abstracts the
> process of data transfer between stages from flink runtime as shuffle
> service. I propose to YARN implementation for flink external shuffle service
> since YARN is widely used in various companies.
> The basic idea is as follows:
> (1) Producers write intermedia result partitions to local disks assigned by
> NodeManager;
> (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary
> service, are acknowledged of intermedia result partition descriptions by
> producers;
> (3) Consumers fetch intermedia result partition from yarn shuffle servers;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
