zhijiangW commented on a change in pull request #10375: [FLINK-14845][runtime]
Introduce data compression to reduce disk and network IO of shuffle.
URL: https://github.com/apache/flink/pull/10375#discussion_r353539452
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/Buffer.java
##########
@@ -217,4 +217,14 @@
* @return self as ByteBuf implementation.
*/
ByteBuf asByteBuf();
+
+ /**
+ * @return whether the buffer is compressed or not.
+ */
+ boolean isCompressed();
+
+ /**
+ * Tags the buffer as compressed or uncompressed.
+ */
+ void setCompressed(boolean isCompressed);
Review comment:
After reviewing the consumer side as I mentioned via
https://github.com/apache/flink/pull/10375/commits/2f538ab20262fbe9981d4851b93e9bee04fc6ad0#r353537910,
It might be feasible to not touch `Buffer` class completely. On producer
side we can judge whether to write compressed buffer via optional
`BufferCompressor` variable in `ResultPartition. On consumer side we can also
judge whether to decompress the buffer via optional `BufferDecompressor`
variable in `SingleInputGate`. And both these optional compressor/decompressor
are determined by global configuration of enabling compression or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
