Re: Checkpointing when one of the sources has completed

[Paul Lam]

Hi Niels,

The link was broken, it should be  
https://issues.apache.org/jira/browse/FLINK-2491 
.

A similar question was asked a few days ago.

Best,
Paul Lam


> 在 2018年10月17日，19:56，Niels van Kaam  写道：
> 
> Hi All,
> 
> Thanks for the responses, the finished source explains my issue then. I can 
> work around the problem by letting my sources negotiate a "final" checkpoint 
> via zookeeper.
> 
> @Paul, I think your answer was meant for the earlier question asked by Joshua?
> 
> Cheers,
> Niels
> 
> On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan  > wrote:
> Hi Niels,
> 
> Probably not, an operator begins to do checkpoint until it gets all the 
> barriers from all the upstream sources, if one source can not send a barrier, 
> the downstream operator can not do checkpoint, FYI.
> 
> Yours sincerely
> Joshua
> 
> On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam  > wrote:
> Hi All,
> 
> I am debugging an issue where the periodic checkpointing has halted. I 
> noticed that one of the sources of my job has completed (finished). The other 
> sources and operators would however still be able to produce output.
> 
> Does anyone know if Flink's periodic checkpoints are supposed to continue 
> when one or more sources of a job are in the "FINISHED" state?
> 
> Cheers,
> Niels
> 

Re: Checkpointing when one of the sources has completed

[Niels van Kaam]

Hi All,

Thanks for the responses, the finished source explains my issue then. I can
work around the problem by letting my sources negotiate a "final"
checkpoint via zookeeper.

@Paul, I think your answer was meant for the earlier question asked by
Joshua?

Cheers,
Niels

On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan  wrote:

> Hi Niels,
>
> Probably not, an operator begins to do checkpoint until it gets all the
> barriers from all the upstream sources, if one source can not send a
> barrier, the downstream operator can not do checkpoint, FYI.
>
> Yours sincerely
> Joshua
>
> On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam  wrote:
>
>> Hi All,
>>
>> I am debugging an issue where the periodic checkpointing has halted. I
>> noticed that one of the sources of my job has completed (finished). The
>> other sources and operators would however still be able to produce output.
>>
>> Does anyone know if Flink's periodic checkpoints are supposed to continue
>> when one or more sources of a job are in the "FINISHED" state?
>>
>> Cheers,
>> Niels
>>
>>

Re: Checkpointing when one of the sources has completed

[Joshua Fan]

Hi Niels,

Probably not, an operator begins to do checkpoint until it gets all the
barriers from all the upstream sources, if one source can not send a
barrier, the downstream operator can not do checkpoint, FYI.

Yours sincerely
Joshua

On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam  wrote:

> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I
> noticed that one of the sources of my job has completed (finished). The
> other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue
> when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>
>

Re: Checkpointing when one of the sources has completed

[Fabian Hueske]

Hi Niels,

Checkpoints can only complete if all sources are running.
That's because the checkpoint mechanism relies on injecting checkpoint
barriers into the stream at the sources.

Best, Fabian

Am Mi., 17. Okt. 2018 um 11:11 Uhr schrieb Paul Lam :

> Hi Niels,
>
> Please see https://issues.apache.org/jira/browse/FLINK-249.
>
> Best,
> Paul Lam
>
> 在 2018年10月17日，16:58，Niels van Kaam  写道：
>
> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I
> noticed that one of the sources of my job has completed (finished). The
> other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue
> when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>
>
>

Re: Checkpointing when one of the sources has completed

[Paul Lam]

Hi Niels,

Please see https://issues.apache.org/jira/browse/FLINK-249 
.

Best,
Paul Lam

> 在 2018年10月17日，16:58，Niels van Kaam  写道：
> 
> Hi All,
> 
> I am debugging an issue where the periodic checkpointing has halted. I 
> noticed that one of the sources of my job has completed (finished). The other 
> sources and operators would however still be able to produce output.
> 
> Does anyone know if Flink's periodic checkpoints are supposed to continue 
> when one or more sources of a job are in the "FINISHED" state?
> 
> Cheers,
> Niels
>