[jira] [Commented] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[Aljoscha Krettek (JIRA)]

    [ 
https://issues.apache.org/jira/browse/BEAM-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16402220#comment-16402220
 ] 

Aljoscha Krettek commented on BEAM-2393:
----------------------------------------

Ah, I think that explains it. This call in the source: 
https://github.com/apache/beam/blob/6148a6d063a0503ee435ab5084fcba3fb864b26f/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java#L333
 will at some point invoke the ParDo because the operations are chained 
together, i.e. they are being executed in the same thread and "sending" 
elements downstream is just method calls. This means that the source will block 
for about 20s in that synchronized block, which keeps the checkpointing logic 
from acquiring the lock.

> BoundedSource is not fault-tolerant in FlinkRunner Streaming mode
> -----------------------------------------------------------------
>
>                 Key: BEAM-2393
>                 URL: https://issues.apache.org/jira/browse/BEAM-2393
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Jingsong Lee
>            Assignee: Grzegorz Kołakowski
>            Priority: Major
>
> {{BoundedSourceWrapper}} does not implement snapshot() and restore(), when 
> the failure to restart, it will send duplicate data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)