[
https://issues.apache.org/jira/browse/FLINK-31655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710956#comment-17710956
]
Anton Kalashnikov commented on FLINK-31655:
-------------------------------------------
Hi [~tartarus] thanks for this discussion.
I think I remember the same discussion which [~gaoyunhaii] mentioned above. I
just want to add that I tried to implement it as well. You can take a look at
it here - https://github.com/apache/flink/pull/16224. My solution works without
any extra synchronization and locks but the performance for other
rebalances(non-Adaptive) was impacted anyway. I changed only
`BufferWritingResultPartition#emitRecord` and added a non-volatile variable to
`PipelinedSubpartition` which was enough for small degradation in the benchmark
since it is a hot path.
I don't remember exactly why we stopped discussing it but we should be careful
about it since Adaptive Rebalance is not so common usage and it will be
unfortunate if we slow down more common rebalances in favor to have this less
common one.
> Adaptive Channel selection for partitioner
> ------------------------------------------
>
> Key: FLINK-31655
> URL: https://issues.apache.org/jira/browse/FLINK-31655
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: tartarus
> Assignee: tartarus
> Priority: Major
>
> In Flink, if the upstream and downstream operator parallelism is not the
> same, then by default the RebalancePartitioner will be used to select the
> target channel.
> In our company, users often use flink to access redis, hbase or other rpc
> services, If some of the Operators are slow to return requests (for external
> service reasons), then because Rebalance/Rescale are Round-Robin the Channel
> selection policy, so the job is easy to backpressure.
> Because the Rebalance/Rescale policy does not care which subtask the data is
> sent to downstream, so we expect Rebalance/Rescale to refer to the processing
> power of the downstream subtask when choosing a Channel.
> Send more data to the free subtask, this ensures the best possible throughput
> of job!
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
