[
https://issues.apache.org/jira/browse/FLINK-31005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691113#comment-17691113
]
Lijie Wang edited comment on FLINK-31005 at 2/20/23 10:02 AM:
--------------------------------------------------------------
I have tested all cases in the description through a simple word count job,
everything met expectations.
1. I tested 6 kind of sinks:
* speculative SinkFunction
* non-speculative SinkFunction
* speculative OutputFormat
* non-speculative OutputFormat
* speculative SinkV2
* non-speculative SinkV2
and let task 0 significantly slower than other tasks by adding sleep in the
code. Finally, in the case of {{speculative SinkFunction}} {{speculative
OutputFormat}} and {{{}speculative SinkV2{}}}, task 0 was detected as a slow
vertex, and Flink created a new speculative execution for it.
2. I ran all the built-in sinks: DiscardingSink, PrintSinkFunction, PrintSink,
FileSink and HiveTableSink (depending on FileSystemOutputFormat), and confirmed
that they all support speculative execution. I checked the results of FileSink
and HiveTableSink, and the results are as expected (same as without speculative
execution).
was (Author: wanglijie95):
I have tested all cases in the description through a simple word count job,
everything met expectations.
1. I tested 6 kind of sinks:
* speculative SinkFunction
* non-speculative SinkFunction
* speculative OutputFormat
* non-speculative OutputFormat
* speculative SinkV2
* non-speculative SinkV2
and let task 0 significantly slower than other tasks by adding sleep in the
code. Finally, in the case of {{speculative SinkFunction}} {{speculative
OutputFormat}} and {{speculative SinkV2}}, task 0 was detected as a slow
vertex, and Flink created a new speculative execution for it.
2. I ran all the built-in sinks: DiscardingSink, PrintSinkFunction, PrintSink,
FileSink and HiveTableSink (depending on FileSystemOutputFormat), and confirmed
that they all support speculative execution. I checked the results of FileSink
and HiveTableSink, and the results are as expected (same as without speculative
execution).
> Release Testing: Verify FLIP-281 Supports speculative execution of sinks
> ------------------------------------------------------------------------
>
> Key: FLINK-31005
> URL: https://issues.apache.org/jira/browse/FLINK-31005
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Zhu Zhu
> Assignee: Lijie Wang
> Priority: Blocker
> Fix For: 1.17.0
>
>
> This task aims to verify [FLIP-281 Supports speculative execution of
> sinks|https://issues.apache.org/jira/browse/FLINK-30725].
> The documentation can be found
> [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#enable-sinks-for-speculative-execution]
> .
> Things to verify:
> 1. If a sink implements the decorative interface
> {{SupportsConcurrentExecutionAttempts}, Speculative executions can be
> performed, otherwise not. Sinks to verify includes SinkFunction, OutputFormat
> and Sink(V2).
> 2. These built-in sinks supports speculative execution: DiscardingSink,
> PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat, HiveTableSink
> It's important to verify that the final data produced are correct and the
> operator chain is as expected, i.e. no unexpected chain-break.
> If it's hard to construct a case that speculative execution would happen,
> especially for those built-in sinks, the speculative execution configuration
> can be tuned to allow it easier to happen, e.g. set
> {{slow-task-detector.execution-time.baseline-lower-bound}} and
> {{slow-task-detector.execution-time.baseline-ratio}} to {{0}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
