[
https://issues.apache.org/jira/browse/FLINK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844642#comment-16844642
]
zhijiang commented on FLINK-10653:
----------------------------------
Thanks for the concerns. [~prudenko]
Actually we ever also researched the RDMA shuffle manager in Spark and the
performance seems really increased much. I think after the basic architecture
of shuffle manager in flink is ready, it could be feasible for expanding other
shuffle implementations like RDMA. We could keep in touch and see then. :)
> Introduce Pluggable Shuffle Manager Architecture
> ------------------------------------------------
>
> Key: FLINK-10653
> URL: https://issues.apache.org/jira/browse/FLINK-10653
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
>
> This is the umbrella issue for improving shuffle architecture.
> Shuffle is the process of data transfer between stages, which involves in
> writing outputs on sender side and reading data on receiver side. In flink
> implementation, it covers three parts of writer, transport layer and reader
> separately which are uniformed for both streaming and batch jobs.
> In detail, the current ResultPartitionWriter interface on upstream side only
> supports in-memory outputs for streaming job and local persistent file
> outputs for batch job. If we extend to implement another writer such as
> DfsWriter, RdmaWriter, SortMergeWriter, etc based on ResultPartitionWriter
> interface, it has not the unified mechanism to extend the reader side
> accordingly.
> In order to make the shuffle architecture more flexible and support more
> scenarios especially for batch jobs, a high level shuffle architecture is
> necessary to manage and extend both writer and reader sides together.
> Refer to the design doc for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
