[
https://issues.apache.org/jira/browse/FLINK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388781#comment-17388781
]
Timo Walther commented on FLINK-23402:
--------------------------------------
{{RuntimeExecutionMode}} cannot be easily changed to a non-enum because it is
deeply integrated into the regular Flink {{ConfigOption}} architecture. Also,
we actually discourage setting it in the {{StreamExecutionEnvironment}} in the
docs:
{code}
We recommend users to NOT set the runtime mode in their program but to instead
set it using the command-line when submitting the application. Keeping the
application code configuration-free allows for more flexibility as the same
application can be executed in any execution mode.
{code}
Furthermore, we need a way of setting in SQL and Table API via string
properties. A unified configuration as mentioned in FLIP-134 therefore makes a
lot of sense.
The {{StreamGraphGenerator}} makes sure that only valid combinations are
supported. Otherwise an exception will be thrown. Esp. for
{{RuntimeExecutionMode.AUTOMATIC}} this is the only location to check the
validity of the ShuffleMode because we don't know whether we execute in batch
or streaming mode until we have scanned all transformations.
I'm fine to change the {{ShuffleMode}} to {{BatchShuffleMode}} and drop
{{AUTOMATIC}} if that helps clarifying concepts
> Expose a consistent GlobalDataExchangeMode
> ------------------------------------------
>
> Key: FLINK-23402
> URL: https://issues.apache.org/jira/browse/FLINK-23402
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
> Reporter: Timo Walther
> Assignee: Timo Walther
> Priority: Major
> Labels: pull-request-available
>
> The Table API makes the {{GlobalDataExchangeMode}} configurable via
> {{table.exec.shuffle-mode}}.
> In Table API batch mode the StreamGraph is configured with
> {{ALL_EDGES_BLOCKING}} and in DataStream API batch mode
> {{FORWARD_EDGES_PIPELINED}}.
> I would vote for unifying the exchange mode of both APIs so that complex SQL
> pipelines behave identical in {{StreamTableEnvironment}} and
> {{TableEnvironment}}. Also the feedback a got so far would make
> {{ALL_EDGES_BLOCKING}} a safer option to run pipelines successfully with
> limited resources.
> [~lzljs3620320]
> {quote}
> The previous history was like this:
> - The default value is pipeline, and we find that many times due to
> insufficient resources, the deployment will hang. And the typical use of
> batch jobs is small resources running large parallelisms, because in batch
> jobs, the granularity of failover is related to the amount of data processed
> by a single task. The smaller the amount of data, the faster the fault
> tolerance. So most of the scenarios are run with small resources and large
> parallelisms, little by little slowly running.
> - Later, we switched the default value to blocking. We found that the better
> blocking shuffle implementation would not slow down the running speed much.
> We tested tpc-ds and it took almost the same time.
> {quote}
> [~dwysakowicz]
> {quote}
> I don't see a problem with changing the default value for DataStream batch
> mode if you think ALL_EDGES_BLOCKING is the better default option.
> {quote}
> In any case, we should make this configurable for DataStream API users and
> make the specific Table API option obsolete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
