[GitHub] flink pull request #5603: [FLINK-8814] Control over the extension of part fi...

[jelmerk]

Github user jelmerk closed the pull request at:

https://github.com/apache/flink/pull/5603


---

[GitHub] flink pull request #5603: [FLINK-8814] Control over the extension of part fi...

[aljoscha]

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5603#discussion_r171553866
  
--- Diff: 
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
 ---
@@ -986,6 +996,14 @@ private void 
handlePendingFilesForPreviousCheckpoints(Map pe
return this;
}
 
+   /**
+* Sets the prefix of part files.  The default is no suffix.
--- End diff --

`prefix` -> `suffix`


---

[GitHub] flink pull request #5603: [FLINK-8814] Control over the extension of part fi...

[jelmerk]

GitHub user jelmerk opened a pull request:

https://github.com/apache/flink/pull/5603

[FLINK-8814] Control over the extension of part files created by 
BucketingSink

## What is the purpose of the change

Popular tools like hue and the avro connector for spark require files 
stored on hdfs to have the .avro extension. This patch makes it possible to 
configure a part file suffix

## Brief change log

- adds support for partSuffix in BucketingSink

## Verifying this change

The basic functionality of BucketingSink is verified by BucketingSinkTest. 
The structure of this test makes it awkward to test this in isolation

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? JavaDocs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jelmerk/flink FLINK_8814

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5603


commit 6487f0fb03870a74b948105ec685462e7b00cbc2
Author: Jelmer Kuperus 
Date:   2018-02-28T20:34:08Z

[FLINK-8814] [file system sinks] Control over the extension of part files 
created by BucketingSink.




---