Hey all, A while ago I proposed an API http://s.apache.org/fileio-write . It has just landed on master https://github.com/apache/beam/pull/3817, in somewhat improved form compared to the initial proposal.
I think it's a cool API and I'm excited that it'll be in Beam 2.3. Please give it a try (e.g. by using 2.3.0-SNAPSHOT) :) Check out some examples in the Javadoc e.g. https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L242 The main selling points are: - It is really really easy to write files of a custom format using this API (example above shows how one could write List<String>'s to CSV with a header). - The API is very Java8-friendly (much more so than the current DynamicDestinations APIs in TextIO/AvroIO, which I would like to deprecate in Beam 2.3) - It gives a common API to use for various file-based IOs that want to get all the fancy features - e.g. https://github.com/apache/beam/pull/4294 shows how to do that with TFRecordIO and XmlIO: they previously didn't have access to features like dynamic destinations, and now they do: you can use TFRecordIO.sink() and XmlIO.sink() with FileIO.write() or writeDynamic(). Thanks to +Reuven Lax <[email protected]> and +Chamikara Jayalath <[email protected]> for reviews.
