[
https://issues.apache.org/jira/browse/FLINK-31025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan updated FLINK-31025:
------------------------------
Description:
This ticket aims for verifying FLINK-30650: Introduce EXPLAIN PLAN_ADVICE to
provide SQL advice.
More details about this feature and how to use it can be found in this
[documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/explain/].
The verification is divided into two parts:
Part I: Verify "EXPLAIN PLAN_ADVICE" can work with different queries under the
streaming mode, such as a single select/insert/statement set w/ or w/o sub-plan
reuse (configured by "table.optimizer.reuse-sub-plan-enabled").
This indicates once specifying the ExplainDetail as "PLAN_ADVICE"
{code:sql}
EXPLAIN PLAN_ADVICE [SELECT ... FROM ...| INSERT INTO ...| EXECUTE STATEMENT
SET BEGIN INSERT INTO ... END]
{code}
You should find the output should be like the following format (note that the
title is changed to "Optimized Physical Plan with Advice")
{code:sql}
== Abstract Syntax Tree ==
...
== Optimized Physical Plan With Advice ==
...
== Optimized Execution Plan ==
...
{code}
The available advice is attached at the end of "== Optimized Physical Plan With
Advice ==", and in front of "== Optimized Execution Plan =="
If switching to batch mode, you should find the "EXPLAIN PLAN_ADVICE" should
throw UnsupportedOperationException as expected.
Part II: Verify the advice content
Write a group aggregate query, and enable/disable the local-global two-phase
configuration, and test the output.
You should find once the following configurations are enabled, you will get the
"no available advice..." output.
{code:java}
table.exec.mini-batch.enabled
table.exec.mini-batch.allow-latency
table.exec.mini-batch.size
table.optimizer.agg-phase-strategy {code}
was:
This ticket aims for verifying FLINK-30650: Adaptive Batch Scheduler should
also work with hybrid shuffle mode.
More details about this feature and how to use it can be found in this
[documentation|xxx].
The verification is divided into two parts:
Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler
Write a simple Flink batch job using hybrid shuffle mode and submit this job.
Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for
batch job, so you do not need other configuration.
Suppose your job's topology like source -> map -> sink, if your cluster have
enough slots, you should find that source and map are running at the same time.
Part II: Verify hybrid shuffle can work with Speculative Execution
Write a Flink batch job using hybrid shuffle mode which has a subtask running
much slower than others (e.g. sleep indefinitely if it runs on a certain host,
the hostname can be retrieved via InetAddress.getLocalHost().getHostName(), or
if its (subtaskIndex + attemptNumer) % 2 == 0)
Modify Flink configuration file to enable speculative execution and tune the
configuration as you like
Submit the job. Checking the web UI, logs, metrics and produced result.
You should find that once a producer task's one subtask finished, all its
consumer tasks can be scheduled in log.
> Release Testing: Verify FLINK-30650 Introduce EXPLAIN PLAN_ADVICE to provide
> SQL advice
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-31025
> URL: https://issues.apache.org/jira/browse/FLINK-31025
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
> Affects Versions: 1.17.0
> Reporter: Jane Chan
> Priority: Blocker
> Fix For: 1.17.0
>
>
> This ticket aims for verifying FLINK-30650: Introduce EXPLAIN PLAN_ADVICE to
> provide SQL advice.
> More details about this feature and how to use it can be found in this
> [documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/explain/].
> The verification is divided into two parts:
> Part I: Verify "EXPLAIN PLAN_ADVICE" can work with different queries under
> the streaming mode, such as a single select/insert/statement set w/ or w/o
> sub-plan reuse (configured by "table.optimizer.reuse-sub-plan-enabled").
> This indicates once specifying the ExplainDetail as "PLAN_ADVICE"
> {code:sql}
> EXPLAIN PLAN_ADVICE [SELECT ... FROM ...| INSERT INTO ...| EXECUTE STATEMENT
> SET BEGIN INSERT INTO ... END]
> {code}
> You should find the output should be like the following format (note that the
> title is changed to "Optimized Physical Plan with Advice")
> {code:sql}
> == Abstract Syntax Tree ==
> ...
>
> == Optimized Physical Plan With Advice ==
> ...
>
>
> == Optimized Execution Plan ==
> ...
> {code}
> The available advice is attached at the end of "== Optimized Physical Plan
> With Advice ==", and in front of "== Optimized Execution Plan =="
> If switching to batch mode, you should find the "EXPLAIN PLAN_ADVICE" should
> throw UnsupportedOperationException as expected.
>
> Part II: Verify the advice content
> Write a group aggregate query, and enable/disable the local-global two-phase
> configuration, and test the output.
> You should find once the following configurations are enabled, you will get
> the "no available advice..." output.
> {code:java}
> table.exec.mini-batch.enabled
> table.exec.mini-batch.allow-latency
> table.exec.mini-batch.size
> table.optimizer.agg-phase-strategy {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
