[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389014#comment-14389014
]
Colin Patrick McCabe edited comment on HADOOP-11552 at 3/31/15 6:35 PM:
------------------------------------------------------------------------
This work is really interesting and I think it could be helpful for us. I
guess my concern is mainly that it seems like a big job. Do you think it makes
sense to do this in a branch? That way we could experiment without the risk of
destabilizing the production RPC code. We could test out some use-cases and
see what the best API was.
I'm also curious if you have taken a look at HADOOP-10300. It seems similar in
some ways.
was (Author: cmccabe):
If the goal is to try things out, then we should have a branch for this work.
That way we won't destabilize the existing RPC code while we're experimenting.
I also think we should have some real-world use of the API before we merge it.
Unit tests are not a real-world use.
> Allow handoff on the server side for RPC requests
> -------------------------------------------------
>
> Key: HADOOP-11552
> URL: https://issues.apache.org/jira/browse/HADOOP-11552
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt,
> HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt
>
>
> An RPC server handler thread is tied up for each incoming RPC request. This
> isn't ideal, since this essentially implies that RPC operations should be
> short lived, and most operations which could take time end up falling back to
> a polling mechanism.
> Some use cases where this is useful.
> - YARN submitApplication - which currently submits, followed by a poll to
> check if the application is accepted while the submit operation is written
> out to storage. This can be collapsed into a single call.
> - YARN allocate - requests and allocations use the same protocol. New
> allocations are received via polling.
> The allocate protocol could be split into a request/heartbeat along with a
> 'awaitResponse'. The request/heartbeat is sent only when there's a request or
> on a much longer heartbeat interval. awaitResponse is always left active with
> the RM - and returns the moment something is available.
> MapReduce/Tez task to AM communication is another example of this pattern.
> The same pattern of splitting calls can be used for other protocols as well.
> This should serve to improve latency, as well as reduce network traffic since
> the keep-alive heartbeat can be sent less frequently.
> I believe there's some cases in HDFS as well, where the DN gets told to
> perform some operations when they heartbeat into the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
