[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491196#comment-15491196
]
Daryn Sharp commented on HADOOP-11552:
--------------------------------------
The patch is interesting (I planted the idea for this type of functionality).
HADOOP-10300 is a bit different. It does allow for a return value but not a
dynamic response. The ipc processing occurs exactly as normal, down through
the engine, back up, response encoded. The difference is the ability to tell
the ipc later "hold on, don't send the response, there's another precondition
that must be satisfied". Edit logging for hdfs.
This lets the processing return nothing, allowing something else to later
encode an arbitrary response.
Patch needs extensive rebasing, but some things we'll need to be careful about:
* ensure encrypted SASL works correctly, there are subtle ordering issues.
* consider how to deal with the static Server methods that depend on a
thread-local for the call.
* how to make the rpc metrics accurate, otherwise processing time becomes
meaningless
> Allow handoff on the server side for RPC requests
> -------------------------------------------------
>
> Key: HADOOP-11552
> URL: https://issues.apache.org/jira/browse/HADOOP-11552
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt,
> HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt
>
>
> An RPC server handler thread is tied up for each incoming RPC request. This
> isn't ideal, since this essentially implies that RPC operations should be
> short lived, and most operations which could take time end up falling back to
> a polling mechanism.
> Some use cases where this is useful.
> - YARN submitApplication - which currently submits, followed by a poll to
> check if the application is accepted while the submit operation is written
> out to storage. This can be collapsed into a single call.
> - YARN allocate - requests and allocations use the same protocol. New
> allocations are received via polling.
> The allocate protocol could be split into a request/heartbeat along with a
> 'awaitResponse'. The request/heartbeat is sent only when there's a request or
> on a much longer heartbeat interval. awaitResponse is always left active with
> the RM - and returns the moment something is available.
> MapReduce/Tez task to AM communication is another example of this pattern.
> The same pattern of splitting calls can be used for other protocols as well.
> This should serve to improve latency, as well as reduce network traffic since
> the keep-alive heartbeat can be sent less frequently.
> I believe there's some cases in HDFS as well, where the DN gets told to
> perform some operations when they heartbeat into the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
