[
https://issues.apache.org/jira/browse/FLINK-31679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707524#comment-17707524
]
Weijie Guo edited comment on FLINK-31679 at 4/1/23 10:06 AM:
-------------------------------------------------------------
Thanks for the proposal, but I'm a bit curious about whether this feature is
really helpful to the vast majority of users. The main reasons are as follows:
- For streaming job, all edges are always {{PIPELINED_BOUNDED}}.
- For batch job, in most cases all edges are {{BLOCKING}}.
- Most users will not use pipelined shuffle in batch scenario, and this may
have a potential scheduling deadlock problem. This is also part of the reason
why we introduced hybrid shuffle.
- Keep this in mind: We plan to make hybrid shuffle the only shuffle
implementation for batch jobs in the next few releases, and will also cover
streaming scenarios in the future.
It seems that the only significance of showing the {{ResultPartitionType}} in
the edge is that in batch job, pipelined shuffle are used, but the planner will
make some of them become blocking. Because hybrid shuffle also supports
scheduling constraints of blocking between upstream and downstream, planer does
not need to introduce additional blocking edges. So this will no longer be the
case in the near future. In most other cases, this information is of little
significance.
was (Author: weijie guo):
Thanks for the proposal, but I'm a bit curious about whether this feature is
really helpful to the vast majority of users. The main reasons are as follows:
- For streaming job, all edges are always {{PIPELINED_BOUNDED}}.
- For batch job, in most cases all edges are {{BLOCKING}}.
- Most users will not use pipelined shuffle in batch scenario, and this may
have a potential scheduling deadlock problem. This is also part of the reason
why we introduced hybrid shuffle.
- Keep this in mind: We plan to make hybrid shuffle the only shuffle
implementation for batch jobs in the next few releases, and will also cover
streaming scenarios in the future.
It seems that the only significance of showing the {{ResultPartitionType}} in
the edge is that in batch job, pipelined shuffles are used, but the planner
will make some of them become blocking. Because hybrid shuffle also supports
scheduling constraints of blocking between upstream and downstream, planer does
not need to introduce additional blocking edges. So this will no longer be the
case in the near future. In most other cases, this information is of little
significance.
> [Flink][UI] Show data exchange type on web ui
> ---------------------------------------------
>
> Key: FLINK-31679
> URL: https://issues.apache.org/jira/browse/FLINK-31679
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
> Affects Versions: 1.18.0
> Reporter: Weihua Hu
> Priority: Major
>
> Flink supports multiple data exchange types (ResultPartitionType), which
> subject to multiple parameters, such as whether job is streaming or batch.
> I think display the data exchange types on the UI would be helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
