GitHub user zhangxinyu1 opened a pull request:
https://github.com/apache/flink/pull/6108
[FLINK-9367] [Streaming Connectors] Allow to do truncate() in calss
BucketingSink when hadoop version is lower than 2.7
## What is the purpose of the change
In the current implementation of class BucketingSink, we cannot use
truncate() function if the hadoop version is lower than 2.7. Instead, it use a
valid-length file to mark how much data is valid.
However, users which reads data from HDFS may not or should not know how
deal with this valid-length file.
Hence, we need a configuration to decide whether use the valid-length file.
If not, we should rewrite the valid file.
## Brief change log
Add a function `enableForceTruncateInProgressFile()` for BucketingSink to
decide whether use the valid-length file. If it's true, the valid-length file
wouldn't be produced. Instead, the valid in-progress file will be rewritten.
## Verifying this change
This change is a trivial work without any test coverage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangxinyu1/flink
force-recovery-file-in-bucketingsink
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6108
----
commit 7c5ba6d54658916e65c40fbbed646efce2c40645
Author: unknown <zhangxinyu@...>
Date: 2018-05-31T12:52:09Z
allow to do truncate() when hadoop version is lower than 2.7
----
---
