akalash commented on a change in pull request #16628:
URL: https://github.com/apache/flink/pull/16628#discussion_r679050435
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/IndexedInputGate.java
##########
@@ -64,4 +64,8 @@ public void blockConsumption(InputChannelInfo channelInfo) {
public void convertToPriorityEvent(int channelIndex, int sequenceNumber)
throws IOException {
getChannel(channelIndex).convertToPriorityEvent(sequenceNumber);
}
+
+ public abstract int getBuffersInUseCount();
Review comment:
It is actually a big question does it make sense to calculate the buffer
size based on `buffers in use` or not. because as you can see the simple
implementation requires extra synchronization + iteration over all channels
which altogether is pretty expensive. It is possible to optimize this solution
but it will lead to extra complexity. I still assume that it should help with
data skew (at least it should be better than the total configured size) but I
don't sure that it is the best tradeoff between performance and implementation
so we need to investigate it somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
