Jiayi-Liao commented on a change in pull request #11567: [FLINK-16645] Limit
the maximum backlogs in subpartitions
URL: https://github.com/apache/flink/pull/11567#discussion_r407302529
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
##########
@@ -375,4 +397,29 @@ void onConsumedSubpartition(int subpartitionIndex) {
private void checkInProduceState() throws IllegalStateException {
checkState(!isFinished, "Partition already finished.");
}
+
+ /**
+ * Check whether all subpartitions' backlogs are less than the
limitation of max backlogs, and make this partition
+ * available again if yes.
+ */
+ public void notifyDecreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel) {
+ if (--unavailableSubpartitionsCount == 0) {
+ CompletableFuture<?> toNotify =
availabilityHelper.getUnavailableToResetAvailable();
+ toNotify.complete(null);
+ }
+ }
+ }
+
+ /**
+ * Check whether any subpartition's backlog exceeds the limitation of
max backlogs, and make this partition
+ * unavailabe if yes.
+ */
+ public void notifyIncreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel + 1) {
+ if (++unavailableSubpartitionsCount == 1) {
+ availabilityHelper.resetUnavailable();
+ }
+ }
+ }
Review comment:
Okay. I'm going to implement this in your way. This will introduce a few
function changes in LocalBufferPool:
1. Interface changes in BufferProvider. Add `subpartitionIndex` as a
parameter to `requestBufferBuilder` and `requestBufferBuilderBlocking` so that
`BufferPool` knows which subpartition the request comes from.
2. Add `requestMemorySegment(int subpartitionIndex)` and `recycle(int
subpartitionIndex)`, and new function will reuse most logic of the original
function, and the original one will be like this:
```
private MemorySegment requestMemorySegment() throws IOException {
return requestMemorySegment(INVALID_SUBPARTITION_INDEX);
}
```
3. Introduce class `SubpartitionBufferRecycler`.
```
class SubpartitionBufferRecycler implements BufferRecycler {
private int subpartitionIndex;
private LocalBufferPool bufferPool;
SubpartitionBufferRecycler(int subpartitionIndex, LocalBufferPool
bufferPool) {
this.subpartitionIndex = subpartitionIndex;
this.bufferPool = bufferPool;
}
@Override
public void recycle(MemorySegment memorySegment) {
bufferPool.recycle(memorySegment, subpartitionIndex);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
