StephanEwen commented on a change in pull request #8880: [FLINK-12986]
[network] Fix instability of BoundedBlockingSubpartition under memory pressure.
URL: https://github.com/apache/flink/pull/8880#discussion_r298113422
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
##########
@@ -157,8 +159,9 @@ private static void initializeBoundedBlockingPartitions(
int i = 0;
try {
for (; i < subpartitions.length; i++) {
- subpartitions[i] = new
BoundedBlockingSubpartition(
- i, parent,
ioManager.createChannel().getPathFile().toPath());
+ final File spillFile =
ioManager.createChannel().getPathFile();
+ subpartitions[i] =
+
BoundedBlockingSubpartition.createWithFileAndMemoryMappedReader(i, parent,
spillFile);
Review comment:
I am not confident enough in any other implementation, yet.
I only added them for our testing and benchmarking.
I am also always unsure about adding too many configurable parts for users.
For API / behavior, we need customizability to fit the application and setup
needs.
For runtime and performance, I tend to favor one opinionated way that works
well across the board. If there is another option that is 10% faster in 10% of
the cases, I am not sure users benefit too much from that, but the added
configuration makes it overall harder to understand and predict.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
