[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954207#comment-16954207 ] lichong commented on FLINK-12574: - [~yitz589] I think it's all right for the flink design about this problem, but I want to know how do you resolve this when you resume your application or reprocess old data from MQ such as kafka. In this case we need to consume the data in kafka from a give offset instead of the offset in the checkpoint or savepoint. Any reply will be appreciated. Thx. > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848360#comment-16848360 ] yitzchak lieberman commented on FLINK-12574: you are right, my bad. closing the bug... > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847354#comment-16847354 ] Aljoscha Krettek commented on FLINK-12574: -- Maybe I understand what you mean now. The part counter is in fact checkpointed, so if you only ever restore from the latest checkpoint then no data will be overwritten. If, however, you restore from a checkpoint (or savepoint) that is not the latest checkpoint (or savepoint) then data will be overwritten. I think this is true for most (if not all) sink implementations. Is this in fact your case? > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847298#comment-16847298 ] Biao Liu commented on FLINK-12574: -- Hi [~yitz589], In my opinion this is more like a customized issue. How about change a base directory every time after resuming the job from some checkpoint which is not the latest checkpoint? Maybe with an incremental id suffix? > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846741#comment-16846741 ] yitzchak lieberman commented on FLINK-12574: its not related to restore from checkpoint as the files get their names from sub task id and part counter, like: part-- so if the application cancelled and starts again from some checkpoint the files will be overwritten. > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846638#comment-16846638 ] Aljoscha Krettek commented on FLINK-12574: -- How do you mean? Can you successfully restore from a checkpoint? Or are you restarting your application from scratch every time? > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844746#comment-16844746 ] yitzchak lieberman commented on FLINK-12574: for me it does not restore from latest checkpoint, when i'm using checkpoint dir in s3 > using sink StreamingFileSink files are overwritten when resuming application > causing data loss > -- > > Key: FLINK-12574 > URL: https://issues.apache.org/jira/browse/FLINK-12574 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.8.0 >Reporter: yitzchak lieberman >Priority: Critical > > when part files are saved to s3 bucket (with bucket assigner) with simple > names such as: > part-0-0 and part-1-2 > restarting or resuming application causes checkpoint id to start from 0 and > old files will be replaced by new part files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
[
https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844740#comment-16844740
]
Aljoscha Krettek commented on FLINK-12574:
--
What do you mean by "restarting or resuming". If everything works correctly,
the {{StreamingFileSink}} should checkpoint the counter that it uses for part
files and new part files should have a higher part index than files that were
already written. Of course, this only works when restoring from the latest
checkpoint/savepoint, otherwise that part counter state will be outdated.
> using sink StreamingFileSink files are overwritten when resuming application
> causing data loss
> --
>
> Key: FLINK-12574
> URL: https://issues.apache.org/jira/browse/FLINK-12574
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.8.0
>Reporter: yitzchak lieberman
>Priority: Critical
>
> when part files are saved to s3 bucket (with bucket assigner) with simple
> names such as:
> part-0-0 and part-1-2
> restarting or resuming application causes checkpoint id to start from 0 and
> old files will be replaced by new part files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
