[
https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380324#comment-17380324
]
ASF GitHub Bot commented on HUDI-2029:
--------------------------------------
rmahindra123 commented on a change in pull request #3128:
URL: https://github.com/apache/hudi/pull/3128#discussion_r669296725
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java
##########
@@ -399,4 +419,47 @@ public int compareTo(ValueMetadata o) {
return Long.compare(this.offsetOfValue, o.offsetOfValue);
}
}
+
+ private static class CompressionHandler implements Serializable {
+ private static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576;
+ private static final int DECOMPRESS_INTERMEDIATE_BUFFER_SIZE = 8192;
+
+ // Caching ByteArrayOutputStreams to avoid recreating it for every
operation
+ private final ByteArrayOutputStream compressBaos;
+ private final ByteArrayOutputStream decompressBaos;
+ private final byte[] decompressIntermediateBuffer;
+
+ CompressionHandler() {
+ compressBaos = new
ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE);
+ decompressBaos = new
ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE);
+ decompressIntermediateBuffer = new
byte[DECOMPRESS_INTERMEDIATE_BUFFER_SIZE];
+ }
+
+ private byte[] compressBytes(final byte[] value) throws IOException {
+ compressBaos.reset();
+ Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
+ DeflaterOutputStream dos = new DeflaterOutputStream(compressBaos,
deflater);
+ try {
+ dos.write(value);
+ } finally {
+ dos.close();
+ deflater.end();
+ }
+ return compressBaos.toByteArray();
+ }
+
+ private byte[] decompressBytes(final byte[] bytes) throws IOException {
+ decompressBaos.reset();
+ InputStream in = new InflaterInputStream(new
ByteArrayInputStream(bytes));
+ try {
+ int len;
+ while ((len = in.read(decompressIntermediateBuffer)) > 0) {
Review comment:
yes, it is just used to read from input stream and write to the output
stream. also ran this by Vinoth, and he is fine with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Implement compression for DiskBasedMap in Spillable Map
> -------------------------------------------------------
>
> Key: HUDI-2029
> URL: https://issues.apache.org/jira/browse/HUDI-2029
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Performance
> Reporter: Rajesh Mahindra
> Assignee: Rajesh Mahindra
> Priority: Major
> Labels: pull-request-available
>
> Implement compression for DiskBasedMap in Spillable Map
> Without compression, DiskBasedMap is causing more spilling to disk than
> RockDb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
