[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315474#comment-14315474
]
Sanjay Radia commented on HADOOP-11552:
---------------------------------------
bq. If we move to an offer-based system like Mesos,
You are mixing layers. SId is talking about the RPC layer. The layer above RPC
such as how Yarn resources are obtained and used will be unaffected.
bq. have the resource manager make outgoing connections to the executors
Making outgoing connections as you suggest is another valid approach. For that
to work well we need client-side async support while this jira is proposing a
server-side "async" (I put async in quotes because in my mind the hand-off is
not asycn-rpc since the rpc client blocks till the work is done).
Another good usecase for this jira is the the write-operations on the NN that
write to the journal. Such operations should be handed off to a worker thread
who writes to the journal and then replies. The original handler-thread goes
back to serving new requests as soon as the hand off is done. If we do this we
could drastically reduce the number of handler threads needed in NN (you
already noted the reduction in handler threads for the other use case).
> Allow handoff on the server side for RPC requests
> -------------------------------------------------
>
> Key: HADOOP-11552
> URL: https://issues.apache.org/jira/browse/HADOOP-11552
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: HADOOP-11552.1.wip.txt
>
>
> An RPC server handler thread is tied up for each incoming RPC request. This
> isn't ideal, since this essentially implies that RPC operations should be
> short lived, and most operations which could take time end up falling back to
> a polling mechanism.
> Some use cases where this is useful.
> - YARN submitApplication - which currently submits, followed by a poll to
> check if the application is accepted while the submit operation is written
> out to storage. This can be collapsed into a single call.
> - YARN allocate - requests and allocations use the same protocol. New
> allocations are received via polling.
> The allocate protocol could be split into a request/heartbeat along with a
> 'awaitResponse'. The request/heartbeat is sent only when there's a request or
> on a much longer heartbeat interval. awaitResponse is always left active with
> the RM - and returns the moment something is available.
> MapReduce/Tez task to AM communication is another example of this pattern.
> The same pattern of splitting calls can be used for other protocols as well.
> This should serve to improve latency, as well as reduce network traffic since
> the keep-alive heartbeat can be sent less frequently.
> I believe there's some cases in HDFS as well, where the DN gets told to
> perform some operations when they heartbeat into the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
