ASF GitHub Bot commented on FLINK-9113:

Github user pnowojski commented on a diff in the pull request:

    --- Diff: 
    @@ -366,6 +384,11 @@ public void 
initializeState(FunctionInitializationContext context) throws Except
                        throw new RuntimeException("Error while creating 
FileSystem when initializing the state of the BucketingSink.", e);
    +           // sync on flush for local file systems
    +           if (localSyncOnFlush && (fs instanceof LocalFileSystem) && 
(writerTemplate instanceof StreamWriterBase)) {
    --- End diff --
    Shouldn't `(writerTemplate instanceof StreamWriterBase)` check be converted 
here into `checState(writerTemplate instanceof StreamWriterBase)` inside the if 
branch, and the same check be extracted and validated whenever user calls:
    `setWriter(...)` or `setLocalSyncOnFlush(...)`? So that non 
`StreamWriterBase` and `localSyncOnFlush = true` would be invalid 
configuration? Otherwise users might experience `wtf` moments when flag is 
being ignored after changing their writer.

> Data loss in BucketingSink when writing to local filesystem
> -----------------------------------------------------------
>                 Key: FLINK-9113
>                 URL: https://issues.apache.org/jira/browse/FLINK-9113
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>            Reporter: Timo Walther
>            Assignee: Timo Walther
>            Priority: Major
> This issue is closely related to FLINK-7737. By default the bucketing sink 
> uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for 
> performance reasons. However, this leads to data loss in case of TaskManager 
> failures when writing to a local filesystem 
> {{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default 
> in local filesystem cases and make it possible to disable this behavior if 
> needed.

This message was sent by Atlassian JIRA

Reply via email to