[
https://issues.apache.org/jira/browse/FLINK-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496524#comment-16496524
]
ASF GitHub Bot commented on FLINK-9367:
---------------------------------------
GitHub user zhangxinyu1 opened a pull request:
https://github.com/apache/flink/pull/6108
[FLINK-9367] [Streaming Connectors] Allow to do truncate() in calss
BucketingSink when hadoop version is lower than 2.7
## What is the purpose of the change
In the current implementation of class BucketingSink, we cannot use
truncate() function if the hadoop version is lower than 2.7. Instead, it use a
valid-length file to mark how much data is valid.
However, users which reads data from HDFS may not or should not know how
deal with this valid-length file.
Hence, we need a configuration to decide whether use the valid-length file.
If not, we should rewrite the valid file.
## Brief change log
Add a function `enableForceTruncateInProgressFile()` for BucketingSink to
decide whether use the valid-length file. If it's true, the valid-length file
wouldn't be produced. Instead, the valid in-progress file will be rewritten.
## Verifying this change
This change is a trivial work without any test coverage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangxinyu1/flink
force-recovery-file-in-bucketingsink
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6108
----
commit 7c5ba6d54658916e65c40fbbed646efce2c40645
Author: unknown <zhangxinyu@...>
Date: 2018-05-31T12:52:09Z
allow to do truncate() when hadoop version is lower than 2.7
----
> Truncate() in BucketingSink is only allowed after hadoop2.7
> -----------------------------------------------------------
>
> Key: FLINK-9367
> URL: https://issues.apache.org/jira/browse/FLINK-9367
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
> Affects Versions: 1.5.0
> Reporter: zhangxinyu
> Priority: Major
>
> When output to HDFS using BucketingSink, truncate() is only allowed after
> hadoop2.7.
> If some tasks failed, the ".valid-length" file is created for the lower
> version hadoop.
> The problem is, if other people want to use the data in HDFS, they must know
> how to deal with the ".valid-length" file, otherwise, the data may be not
> exactly-once.
> I think it's not convenient for other people to use the data. Why not just
> read the in-progress file and write a new file when restoring instead of
> writing a ".valid-length" file.
> In this way, others who use the data in HDFS don't need to know how to deal
> with the ".valid-length" file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
