zentol commented on a change in pull request #8654: [FLINK-12647][network] Add
feature flag to disable release of consumed blocking partitions
URL: https://github.com/apache/flink/pull/8654#discussion_r292826430
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/ResultPartitionDeploymentDescriptor.java
##########
@@ -58,6 +72,7 @@ public ResultPartitionDeploymentDescriptor(
KeyGroupRangeAssignment.checkParallelismPreconditions(maxParallelism);
this.maxParallelism = maxParallelism;
this.sendScheduleOrUpdateConsumersMessage =
sendScheduleOrUpdateConsumersMessage;
+ this.isManagedExternally =
partitionDescriptor.getPartitionType() == ResultPartitionType.BLOCKING;
Review comment:
So, my thinking is that the ResultPartitionType has no bearing on whether a
partition can be consumed multiple times or is externally managed. These are
all implementation details of the ShuffleEnvironment.
The only thing that the type itself specifies is whether or not it is
consumable while it is being produced.
Now, so far we _did_ couple these things very explicitly to the type being
BLOCKING (hence why this field is also based on the type), but long-term this
might be problematic.
For example, we may at some point want to make pipelined partitions
re-consumable as well, by also spilling them to disk while streaming them to
producers at the same time.
If we bake this property into the type we'd have to add additional
ResultPartitionTypes in the future to allow this to be done, which typically is
a tricky think to do.
Additionally, the scheduler (at a later point) may decide that a certain
blocking partition should not be managed by the JM but by the environment,
which a separate flag would allow.
(At which point we'D have to modify the constructor obviously)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
