[
https://issues.apache.org/jira/browse/FLINK-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366763#comment-16366763
]
Piotr Nowojski commented on FLINK-8625:
---------------------------------------
Yes. Instead of this OutputFlasher thread we want to move this job to
{{PartitionRequestQueue}} inside Netty's thread pool.
\{{PartitionRequestQueue}} could just constantly "busy loop" over all of the
subpartitions and if it encounters an "empty" subpartition, it could back
of/try again later with some timeout. This will have following advantages:
# No GC issues with constant and frequent scheduling data notification caused
by current OutputFlusher thread. If we have 1000 output channels and 1ms flush
timeout this adds up to 1 000 000 notifications per second.
# No context switching and no synchronisation for the flushing
# Instead of notifying 1000 output channels one by one as it's happening right
now, one thread running one instance of \{{PartitionRequestQueue}} will handle
all of the assigned to him output channels - with 10 Netty threads, each thread
will handle ~100 output channels in one flush.
# In case of full throughput scenario when all partitions are constantly full,
we can skip the flush.
> Move OutputFlusher thread to Netty scheduled executor
> -----------------------------------------------------
>
> Key: FLINK-8625
> URL: https://issues.apache.org/jira/browse/FLINK-8625
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Reporter: Piotr Nowojski
> Priority: Major
>
> This will allow us to trigger/schedule next flush only if we are not
> currently busy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
