zhijiangW commented on a change in pull request #10375: [FLINK-14845][runtime]
Introduce data compression to reduce disk and network IO of shuffle.
URL: https://github.com/apache/flink/pull/10375#discussion_r354104779
##########
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannelTest.java
##########
@@ -431,8 +426,35 @@ public void testGetNextAfterPartitionReleased() throws
Exception {
assertFalse(channel.getNextBuffer().isPresent());
}
+ /**
+ * Verifies that buffer is not compressed when getting from a {@link
LocalInputChannel}.
+ */
+ @Test
+ public void testGetBufferFromLocalChannelWhenCompressionEnabled()
throws Exception {
+ ResultSubpartitionView subpartitionView =
createResultSubpartitionView(true);
+ TestingResultPartitionManager partitionManager = new
TestingResultPartitionManager(subpartitionView);
+ LocalInputChannel channel = createLocalInputChannel(new
SingleInputGateBuilder().build(), partitionManager);
+
+ // request partition and get next buffer
+ channel.requestSubpartition(0);
+ Optional<InputChannel.BufferAndAvailability>
bufferAndAvailability = channel.getNextBuffer();
+ assertTrue(bufferAndAvailability.isPresent());
+
assertFalse(bufferAndAvailability.get().buffer().isCompressed());
+ }
+
//
---------------------------------------------------------------------------------------------
+ private static ResultSubpartitionView
createResultSubpartitionView(boolean addBuffer) throws IOException {
+ int bufferSize = 4096;
+ ResultPartition parent = PartitionTestUtils.createPartition(
+ ResultPartitionType.PIPELINED,
NoOpFileChannelManager.INSTANCE, true, "LZ4", bufferSize);
+ ResultSubpartition subpartition = new PipelinedSubpartition(0,
parent);
+ if (addBuffer) {
+
subpartition.add(BufferBuilderTestUtils.createFilledFinishedBufferConsumer(bufferSize));
Review comment:
Do not need to create `subpartition` above and use
`parent.getAllPartitions()[0]` instead, then we do not change the package
public for `PipelinedSubpartition` class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
