[
https://issues.apache.org/jira/browse/FLINK-31655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711755#comment-17711755
]
Gen Luo commented on FLINK-31655:
---------------------------------
Hi [~akalash], thanks for the remind.
I agree that the impaction to the normal rebalance must be measured. Though I
wonder why the implementation in the pull request could impact significantly.
Hopefully we can implement this time in a more controllable way and make sure
the impaction is acceptable.
On the other hand, in my opinion, if the adaptive rebalance does little
regression to the performance, maybe we can make it 'suggested' or even
'default', unless the users need all subtasks process exactly the same amount
of records.
Hi [~tartarus],
Thanks for the doc! As we discussed offline, we'd better make the issue a FLIP
and raise a formal discussion in the mailing list. As others mentioned, the
feature was discussed before and faced quite some problems. I think we need a
formal proposal with some implementation plans, then provide some benchmark
results with a POC version, which include the gain of performance in applicable
scenes, and the impaction to normal rebalance.
> Adaptive Channel selection for partitioner
> ------------------------------------------
>
> Key: FLINK-31655
> URL: https://issues.apache.org/jira/browse/FLINK-31655
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: tartarus
> Assignee: tartarus
> Priority: Major
>
> In Flink, if the upstream and downstream operator parallelism is not the
> same, then by default the RebalancePartitioner will be used to select the
> target channel.
> In our company, users often use flink to access redis, hbase or other rpc
> services, If some of the Operators are slow to return requests (for external
> service reasons), then because Rebalance/Rescale are Round-Robin the Channel
> selection policy, so the job is easy to backpressure.
> Because the Rebalance/Rescale policy does not care which subtask the data is
> sent to downstream, so we expect Rebalance/Rescale to refer to the processing
> power of the downstream subtask when choosing a Channel.
> Send more data to the free subtask, this ensures the best possible throughput
> of job!
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
