zhijiangW commented on a change in pull request #10375: [FLINK-14845][runtime]
Introduce data compression to reduce disk and network IO of shuffle.
URL: https://github.com/apache/flink/pull/10375#discussion_r354099967
##########
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/BoundedDataTestBase.java
##########
@@ -160,27 +181,40 @@ private static void testGetSize(BoundedData bd) throws
Exception {
// utils
//
------------------------------------------------------------------------
- private static int writeInts(BoundedData bd, int numInts) throws
IOException {
+ private static int writeInts(BoundedData bd, int numInts, boolean
compressedEnabled) throws IOException {
final int numIntsInBuffer = BUFFER_SIZE / 4;
int numBuffers = 0;
for (int nextValue = 0; nextValue < numInts; nextValue +=
numIntsInBuffer) {
Buffer buffer =
BufferBuilderTestUtils.buildBufferWithAscendingInts(BUFFER_SIZE,
numIntsInBuffer, nextValue);
- bd.writeBuffer(buffer);
+ if (compressedEnabled) {
+
bd.writeBuffer(COMPRESSOR.compressToInternalBuffer(buffer));
+ } else {
+ bd.writeBuffer(buffer);
+ }
numBuffers++;
}
return numBuffers;
}
- private static void readInts(BoundedData.Reader reader, int
numBuffersExpected, int numInts) throws IOException {
+ private static void readInts(
+ BoundedData.Reader reader,
+ int numBuffersExpected,
+ int numInts,
+ boolean compressedEnabled) throws IOException {
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
