johnjcasey commented on code in PR #26925:
URL: https://github.com/apache/beam/pull/26925#discussion_r1210691804
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java:
##########
@@ -240,8 +244,66 @@ public WritableByteChannel
writeCompressed(WritableByteChannel channel) throws I
return Channels.newChannel(
new SnappyCompressorOutputStream(Channels.newOutputStream(channel),
uncompressedSize));
}
+ },
+
+ /** Four MC Compression Splittable compressed file format leveraging LZ4
+ * Uses LZ4 Fast Compression*/
+ MC4_FAST(".4mc", ".4mc") {
Review Comment:
I believe we will have problems with detection with all of these having the
same suffix configurations
##########
sdks/java/core/build.gradle:
##########
@@ -82,6 +83,7 @@ dependencies {
implementation library.java.antlr_runtime
implementation library.java.commons_compress
implementation library.java.commons_lang3
+ implementation group: 'com.fing.fourmc', name: 'hadoop-4mc', version: '3.0.0'
Review Comment:
Please add this as a library, similar to the other libraries we import
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java:
##########
@@ -240,8 +244,66 @@ public WritableByteChannel
writeCompressed(WritableByteChannel channel) throws I
return Channels.newChannel(
new SnappyCompressorOutputStream(Channels.newOutputStream(channel),
uncompressedSize));
}
+ },
+
+ /** Four MC Compression Splittable compressed file format leveraging LZ4
+ * Uses LZ4 Fast Compression*/
+ MC4_FAST(".4mc", ".4mc") {
+ @Override
+ public WritableByteChannel writeCompressed(WritableByteChannel channel)
throws IOException {
+ return Channels.newChannel(
+ new
FourMcCodec().createOutputStream(Channels.newOutputStream(channel)));
+ }
+
+ @Override
+ public ReadableByteChannel readDecompressed(ReadableByteChannel channel) {
+ return channel;
+ }
+ },
+ /** Four MC Compression Splittable compressed file format leveraging LZ4
+ * Uses LZ4 MC Compression*/
+ MC4_MEDIUM(".4mc", ".4mc") {
Review Comment:
Please add tests for these compression formats. See CompressedSourceTest for
similar tests.
##########
sdks/java/core/build.gradle:
##########
@@ -17,6 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
+apply plugin: 'maven-publish'
Review Comment:
Do you need this here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
