[
https://issues.apache.org/jira/browse/HADOOP-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552617#comment-13552617
]
Luke Lu commented on HADOOP-9194:
---------------------------------
For specific use cases, especially for configurations you can control, a
separate unix domain socket could be a reasonable hack. That said if we have a
nonblocking RPC reader implementation we can do a better job than OS accept
backlog. In general, we don't want to have any queues that we cannot
control/influence.
This actual brings up a serious security issue with the current RPC
implemenation, it's trivial for any (low bandwidth) client to DoS any Hadoop
RPC service (even with unlimited bandwidth) either deliberately or by accident.
In order to fix this critical issue we need to have nonblocking readers. As
Binglin pointed out on HADOOP-9151, the current protobuf RPC protocol is not
amenable to nonblocking implementations.
I propose that we fix this here once for all as well:
{code}
request ::= <request-envelope> <request-protobuf-payload>
request-envelope ::= 'HREQ' <service-class-int8>
<request-protobuf-payload-length-vint32>
response ::= <response-envelope> <response-protobuf-payload>
response-envelope ::= 'HRES' <service-class-int8>
<response-protobuf-payload-length-vint32>
{code}
The new envelopes make nonblocking network IO trivial for rpc
server/proxy/switches. The 'magic' 4-bytes makes debugging tcpdump and/or
adding support to wireshark easier as well.
> RPC Support for QoS
> -------------------
>
> Key: HADOOP-9194
> URL: https://issues.apache.org/jira/browse/HADOOP-9194
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 2.0.2-alpha
> Reporter: Luke Lu
>
> One of the next frontiers of Hadoop performance is QoS (Quality of Service).
> We need QoS support to fight the inevitable "buffer bloat" (including various
> queues, which are probably necessary for throughput) in our software stack.
> This is important for mixed workload with different latency and throughput
> requirements (e.g. OLTP vs OLAP, batch and even compaction I/O) against the
> same DFS.
> Any potential bottleneck will need to be managed by QoS mechanisms, starting
> with RPC.
> How about adding a one byte DS (differentiated services) field (a la the
> 6-bit DS field in IP header) in the RPC header to facilitate the QoS
> mechanisms (in separate JIRAs)? The byte at a fixed offset (how about 0?) of
> the header is helpful for implementing high performance QoS mechanisms in
> switches (software or hardware) and servers with minimum decoding effort.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
