[
https://issues.apache.org/jira/browse/FLINK-35690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17861325#comment-17861325
]
Yuxin Tan commented on FLINK-35690:
-----------------------------------
[~fanrui] Thanks for the reminding. We have assigned this Jira to [~yyhx] and
he will help test it.
> Release Testing: Verify FLIP-459: Support Flink hybrid shuffle integration
> with Apache Celeborn
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-35690
> URL: https://issues.apache.org/jira/browse/FLINK-35690
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Yuxin Tan
> Assignee: xuhuang
> Priority: Blocker
> Labels: release-testing
> Fix For: 1.20.0
>
>
> Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533
> In Flink 1.20, we proposed integrating Flink's Hybrid Shuffle with Apache
> Celeborn through a pluggable remote tier interface. To verify this feature,
> you should reference these main two steps.
> 1. Implement Celeborn tier.
> * Implement a new tier factory and tier for Celeborn, including these APIs,
> including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent.
> * The implementations should support granular data management at the Segment
> level for both client and server sides.
> 2. Use the implemented tier to shuffle data.
> * Compile Flink and Celeborn.
> * Deploy Celeborn service
> ** Deploy a new Celeborn service with the new compiled packages. You can
> reference the doc ([https://celeborn.apache.org/docs/latest/]) to deploy the
> cluster.
> * Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink
> classpath.
> * Configure the options to enable the feature.
> ** Configure the option
> taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the
> new Celeborn tier classes. Except for this option, the following options
> should also be added.
> {code:java}
> execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL
> celeborn.master.endpoints: <the celeborn endpoint address>
> celeborn.client.shuffle.partition.type: MAP{code}
> * Run some test examples(e.g., WordCount) to verify the feature.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
