Aljoscha Krettek updated FLINK-9367:
    Fix Version/s:     (was: 1.5.0)

> Truncate() in BucketingSink is only allowed after hadoop2.7
> -----------------------------------------------------------
>                 Key: FLINK-9367
>                 URL: https://issues.apache.org/jira/browse/FLINK-9367
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming Connectors
>    Affects Versions: 1.5.0
>            Reporter: zhangxinyu
>            Priority: Major
> When output to HDFS using BucketingSink, truncate() is only allowed after 
> hadoop2.7.
> If some tasks failed, the ".valid-length" file is created for the lower 
> version hadoop.
> The problem is, if other people want to use the data in HDFS, they must know 
> how to deal with the ".valid-length" file, otherwise, the data may be not 
> exactly-once.
> I think it's notĀ convenient for other people to use the data. Why not just 
> read the in-progress file and write a new file when restoring instead of 
> writing a ".valid-length" file.
> In this way, others who use the data in HDFS don't need to know how to deal 
> with the ".valid-length" file.

This message was sent by Atlassian JIRA

Reply via email to