meetdestiny commented on PR #17599:
URL: https://github.com/apache/beam/pull/17599#issuecomment-1121849505
Agree. 4mc is a file format but the codec worked for us using TextIO.
This is a sample pipeline which works with 4mc file format.
PCollection<String> collection = p
.apply("Read " + topic + " Messages",
PubsubIO.readStrings().fromTopic(topicName))
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(1))));
collection.apply(FileIO.<String,
String>writeDynamic().withNumShards(1)
.by((SerializableFunction<String, String>) input ->
input)
.via(TextIO.sink()).withCompression(Compression.MC4).to(options.getBucket()+"/output/")
.withDestinationCoder(StringUtf8Coder.of())
.withNaming( new SerializableFunction<String,
FileNaming> () {
@Override
public FileNaming apply(String input) {
return (window, pane, numShards, shardIndex,
compression) -> {
return input.toString() +".4mc";
};
}
}));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
