[
https://issues.apache.org/jira/browse/FLINK-11035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718897#comment-16718897
]
Piotr Nowojski commented on FLINK-11035:
----------------------------------------
[~zjwang] Have you observed this to be a problem in the real world? Problem is
that any kind of notifications in this area can be quite costly, however now
I'm not sure if I was benchmarking fix for that or not (I would guess yes, but
again - I'm not sure). When I was implementing this I was aware about this
scenario that you are describing, but it seemed like a non-issue - we can
always relay on output flushing to do the trick. After all this only affects
scenarios where:
1. we are not under full throughput, because we are blocked on requesting next
buffer
2. it only increases the latency for records in rare cases when buffer is full
after serialising next record. In other words, if record i 320bytes long, this
extra notification will only help for 1 record out of 100.
> Notify data available to network stack immediately after finishing
> BufferBuilder
> --------------------------------------------------------------------------------
>
> Key: FLINK-11035
> URL: https://issues.apache.org/jira/browse/FLINK-11035
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> The data availability notification for network relies on whether there are
> finished _BufferBuilder_ or flush triggered. If flush is not triggered and
> the first _BufferBuilder_ enqueues into the subpartition, although this
> _BufferBuilder_ is finished on _RecordWriter_ side, it has to rely on
> enqueuing the second _BufferBuilder_ to trigger notification available. It
> may bring some delays for transporting the finished _BufferBuilder_ in
> network, especially there has a blocking operation for requesting the second
> _BufferBuilder_ from pool.
> Supposing there is only one available buffer in LocalBufferPool in extreme
> scenarios, if the first _BufferBuilder_ is not transported and recycled, the
> requesting for second _BufferBuilder_ will be blocked all the time.
> I propose to add a _notifyBufferFinished_ method in _ResultPartitionWriter_
> interface, then _RecordWriter_ can notify via it after _BufferBuilder_
> finished_._
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
