[
https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325814#comment-17325814
]
Guowei Ma commented on FLINK-13247:
-----------------------------------
I am very glad that you are interested in this topic. [~wind_ljy]
We also implemented a Remote Shuffle Service based on 1.13, this is mainly
taking into account the increasingly common situation of storage and computing
separation and containerization. At present, it is mainly deployed on k8s, of
course, it is not a big problem to deploy to yarn. In the process of
implementation, we also found some minor problems with pluggable shuffle
service architecture, and we plan to initiate some discussions in the community.
If you are also interested, we can take time to chat offline.
> Implement external shuffle service for YARN
> -------------------------------------------
>
> Key: FLINK-13247
> URL: https://issues.apache.org/jira/browse/FLINK-13247
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: MalcolmSanders
> Priority: Major
>
> Flink batch job users could achieve better cluster utilization and job
> throughput throught external shuffle service because the producers of
> intermedia result partitions can be released once intermedia result
> partitions have been persisted on disks. In
> [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang]
> has introduced pluggable shuffle manager architecture which abstracts the
> process of data transfer between stages from flink runtime as shuffle
> service. I propose to YARN implementation for flink external shuffle service
> since YARN is widely used in various companies.
> The basic idea is as follows:
> (1) Producers write intermedia result partitions to local disks assigned by
> NodeManager;
> (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary
> service, are acknowledged of intermedia result partition descriptions by
> producers;
> (3) Consumers fetch intermedia result partition from yarn shuffle servers;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
