zhijiangW commented on a change in pull request #8648: [FLINK-12738][network]
Remove abstract getPageSize method from InputGate
URL: https://github.com/apache/flink/pull/8648#discussion_r298017736
##########
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/InputProcessorUtil.java
##########
@@ -53,16 +54,17 @@ public static CheckpointBarrierHandler
createCheckpointBarrierHandler(
+ " must be positive or -1 (infinite)");
}
+ int pageSize =
ConfigurationParserUtils.getPageSize(taskManagerConfig);
if
(taskManagerConfig.getBoolean(NettyShuffleEnvironmentOptions.NETWORK_CREDIT_MODEL))
{
barrierHandler = new BarrierBuffer(
inputGate,
- new
CachedBufferBlocker(inputGate.getPageSize()),
+ new CachedBufferBlocker(pageSize),
Review comment:
For the case of `BufferSpiller` which uses the page size for requesting
buffer to fill in disk data. Actually this page size is not relevant with the
network buffer size. The default 32K network buffer is only suitable for
network transport. But for reading disk data it should not be limited to the
default 32K network buffer size. Of course it could simply take this size as
default, but I think it has no implicit assumption concern here. Just like
current `IOManager` and `MemoryManager` are both using segment size from the
same config option.
The page size for shuffle service should not be exposed to outside, because
it is only used for internal buffer pool and network transport. For other
shuffle implementations like DfsShuffleService, it might not need the network
buffer pool and no page size at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
