[
https://issues.apache.org/jira/browse/FLINK-31655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738546#comment-17738546
]
tartarus commented on FLINK-31655:
----------------------------------
hi [~fanrui] thank you for your reply, I will answer your concerns one by one
{color:#c1c7d0}_>What's the difference between solution 1(Global Optimal) and
optimized solution 1(Most of the best)?_{color}
{color:#172b4d}the solution 1 (Global Optimal) is Each record of data is
traversed through all subpartitions to find the best channel. So the more
subpartitions, the more time it takes!{color}
{color:#172b4d}_optimized solution 1(Most of the best) is When a record is
comes in, suppose it should be written to subpartition x, and if index x is in
the first half of the subpartition array, then we find the optimal subpartition
index from x to the end of subpartition array; if index x is in the second half
of the subpartition array, then we find the optimal subpartition index from 0
to x;_ {color}
{color:#c1c7d0}_>When maxTraverseSize > the subpartition number, the
maxTraverseSize should be the subpartition number. For example,
maxTraverseSize = 20, and the subpartition number is 10._{color}
Yes, I have taken this case into account when testing
{color:#c1c7d0}_>How the SQL job to use the adaptiveRebalance? The rebalance
should be the default._{color}
If we introduce the new api adaptiveRebalance, how will SQL jobs use it and
will SQL jobs not support adaptiveRebalance for now?
I will refer to everyone's comments to complete the FLIP
> Adaptive Channel selection for partitioner
> ------------------------------------------
>
> Key: FLINK-31655
> URL: https://issues.apache.org/jira/browse/FLINK-31655
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: tartarus
> Assignee: tartarus
> Priority: Major
>
> In Flink, if the upstream and downstream operator parallelism is not the
> same, then by default the RebalancePartitioner will be used to select the
> target channel.
> In our company, users often use flink to access redis, hbase or other rpc
> services, If some of the Operators are slow to return requests (for external
> service reasons), then because Rebalance/Rescale are Round-Robin the Channel
> selection policy, so the job is easy to backpressure.
> Because the Rebalance/Rescale policy does not care which subtask the data is
> sent to downstream, so we expect Rebalance/Rescale to refer to the processing
> power of the downstream subtask when choosing a Channel.
> Send more data to the free subtask, this ensures the best possible throughput
> of job!
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
