MalcolmSanders created FLINK-13247:
--------------------------------------
Summary: Implement external shuffle service for YARN
Key: FLINK-13247
URL: https://issues.apache.org/jira/browse/FLINK-13247
Project: Flink
Issue Type: New Feature
Components: Runtime / Network
Reporter: MalcolmSanders
Assignee: MalcolmSanders
Flink batch job users could achieve better cluster utilization and job
throughput throught external shuffle service because the producers of
intermedia result partitions can be released once intermedia result partitions
have been persisted on disks. In
[FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang] has
introduced pluggable shuffle manager architecture which abstracts the process
of data transfer between stages from flink runtime as shuffle service. I
propose to YARN implementation for flink external shuffle service since YARN is
widely used in various companies.
The basic idea is as follows:
(1) Producers write intermedia result partitions to local disks assigned by
NodeManager;
(2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary service,
are acknowledged of intermedia result partition descriptions by producers;
(3) Consumers fetch intermedia result partition from yarn shuffle servers;
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
