[ 
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428297#comment-16428297
 ] 

ASF GitHub Bot commented on FLINK-9113:
---------------------------------------

Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5811#discussion_r179743870
  
    --- Diff: 
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
 ---
    @@ -366,6 +384,11 @@ public void 
initializeState(FunctionInitializationContext context) throws Except
                        throw new RuntimeException("Error while creating 
FileSystem when initializing the state of the BucketingSink.", e);
                }
     
    +           // sync on flush for local file systems
    +           if (localSyncOnFlush && (fs instanceof LocalFileSystem) && 
(writerTemplate instanceof StreamWriterBase)) {
    --- End diff --
    
    Shouldn't `(writerTemplate instanceof StreamWriterBase)` check be converted 
here into `checState(writerTemplate instanceof StreamWriterBase)` inside the if 
branch, and the same check be extracted and validated whenever user calls:
    `setWriter(...)` or `setLocalSyncOnFlush(...)`? So that non 
`StreamWriterBase` and `localSyncOnFlush = true` would be invalid 
configuration? Otherwise users might experience `wtf` moments when flag is 
being ignored after changing their writer.


> Data loss in BucketingSink when writing to local filesystem
> -----------------------------------------------------------
>
>                 Key: FLINK-9113
>                 URL: https://issues.apache.org/jira/browse/FLINK-9113
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>            Reporter: Timo Walther
>            Assignee: Timo Walther
>            Priority: Major
>
> This issue is closely related to FLINK-7737. By default the bucketing sink 
> uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for 
> performance reasons. However, this leads to data loss in case of TaskManager 
> failures when writing to a local filesystem 
> {{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default 
> in local filesystem cases and make it possible to disable this behavior if 
> needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to