[
https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591251#comment-16591251
]
Fabian Hueske commented on FLINK-10205:
---------------------------------------
Most jobs are implemented in a way that the split assignment does not affect
the semantics of the job.
I don't think we should restrict input split assignment to make the small set
of jobs with non-deterministic logic behave deterministic.
Also, there are a few more sources of non-determinism (order, partitioning)
that would need to be removed. The orchestration overhead to make all
operations behave determistically is too high.
You can implement a custom
[InputSplitAssigner|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java]
if you want to have deterministic input split assignment.
I would close this issue as "Won't Fix"
> Batch Job: InputSplit Fault tolerant for DataSourceTask
> -------------------------------------------------------
>
> Key: FLINK-10205
> URL: https://issues.apache.org/jira/browse/FLINK-10205
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager
> Reporter: JIN SUN
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Today DataSource Task pull InputSplits from JobManager to achieve better
> performance, however, when a DataSourceTask failed and rerun, it will not get
> the same splits as its previous version. this will introduce inconsistent
> result or even data corruption.
> Furthermore, if there are two executions run at the same time (in batch
> scenario), this two executions should process same splits.
> we need to fix the issue to make the inputs of a DataSourceTask
> deterministic. The propose is save all splits into ExecutionVertex and
> DataSourceTask will pull split from there.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
