[
https://issues.apache.org/jira/browse/FLINK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209682#comment-17209682
]
Till Rohrmann commented on FLINK-10653:
---------------------------------------
[~zjwang] should we close this issue and make the open issues follow up issues?
> Introduce Pluggable Shuffle Service Architecture
> ------------------------------------------------
>
> Key: FLINK-10653
> URL: https://issues.apache.org/jira/browse/FLINK-10653
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: Zhijiang
> Assignee: Zhijiang
> Priority: Major
>
> This is the umbrella issue for improving shuffle architecture.
> Shuffle is the process of data transfer between stages, which involves in
> writing outputs on sender side and reading data on receiver side. In flink
> implementation, it covers three parts of writer, transport layer and reader
> separately which are uniformed for both streaming and batch jobs.
> In detail, the current ResultPartitionWriter interface on upstream side only
> supports in-memory outputs for streaming job and local persistent file
> outputs for batch job. If we extend to implement another writer such as
> DfsWriter, RdmaWriter, SortMergeWriter, etc based on ResultPartitionWriter
> interface, it has not the unified mechanism to extend the reader side
> accordingly.
> In order to make the shuffle architecture more flexible and support more
> scenarios especially for batch jobs, a high level shuffle architecture is
> necessary to manage and extend both writer and reader sides together.
> Refer to the design doc for more details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
