Nico Kruber created FLINK-12538:
-----------------------------------
Summary: Network notifyDataAvailable() only called after getting a
new buffer
Key: FLINK-12538
URL: https://issues.apache.org/jira/browse/FLINK-12538
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.8.0, 1.7.2, 1.6.3, 1.9.0
Reporter: Nico Kruber
There is a potential regression in Flink 1.5+ which came with the low-latency
changes. Whenever the {{RecordWriter}} finishes a buffer, it will first ask for
a new buffer, then adds it to the appropriate result subpartition which
notifies Netty of data being available.
In back-pressured scenarios where all buffers from the local pool are taken, it
may happen that you do not immediately get a new buffer and have to wait for as
long as it takes to get it before Netty can make use of the finished network
buffer. Pre 1.5, Flink always immediately notified the downwards stack.
Although we do still have the output flusher notifying Netty within at most
100ms (by default), the new behaviour may actually decrease throughput and
latency in a back-pressured scenario.
Having a quick look at the code, changing this behaviour is probably not too
difficult but only needs to take care not to introduce additional locking /
locking multiple times compared to now. Things to do/consider:
* {{PipelinedSubpartition#add()}} contains some optimisations to avoid
unnecessary flushes but these conditions are under a lock -> try to not acquire
it twice
* {{RecordWriter#requestNewBufferBuilder()}} could therefore maybe have an
optimised path with a non-blocking buffer builder request if successful and if
not, notify/flush and do another blocking request
After talking to [~pnowojski] offline, we are not sure how grave the issue is
and whether we would improve by changing it. If you are willing to take a look
and have code changing the current behaviour, please verify that it does not
cause any performance regression itself and actually does improve some scenario
(shown by a performance test, e.g. via
https://github.com/dataArtisans/flink-benchmarks ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
