Sam Whittle created BEAM-12118:
----------------------------------
Summary: QueuingBeamFnDataClient adds polling latency to
completing bundle processing
Key: BEAM-12118
URL: https://issues.apache.org/jira/browse/BEAM-12118
Project: Beam
Issue Type: Bug
Components: java-fn-execution
Reporter: Sam Whittle
Assignee: Sam Whittle
Currently the inboundDataClients are registered with recieve, and they add data
to a queue. There is no explicit indication from the clients that they are no
longer going to add values to the queue.
Within QueueingBeamFnDataClient.drainAndBlock the queue is therefore polled and
if nothing is present all clients are polled to see if they are complete.
This design makes for unfortunate tradeoffs on poll timeout:
- cpu wasted with small timeout
- additional latency in noticing we have completed with larger timeout
With the existing InboundDataClient interface, we could have a separate thread
call awaitCompletion on all of the clients and then shutdown the queue (adding
a poison pill perhaps)
Or we could modify InboundDataClient interface to allow registering iterest in
when the client is done producing elements. The existing clients all seem
based upon futures which allow that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
