[
https://issues.apache.org/jira/browse/HADOOP-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551391#comment-13551391
]
Luke Lu commented on HADOOP-9194:
---------------------------------
You can use port to differentiate services for IP connections, which
essentially communicates the service class out of band by convention. This is a
reasonable hack for internal use (a la HDFS-599) due to lack of support in the
RPC itself. Things quickly get out of hand if we have more service classes
and/or different transport mechanisms without ports (say, again, unix domain
socket (use another file naming convention?)) let alone support for proxy, load
balancing and firewalls.
If we want use Hadoop RPC as general purpose DFS (or computing) client
protocols, it needs to support QoS natively. Having well defined QoS semantics
in the RPC also lends to common libraries of QoS algorithms that can be easily
adopted at every necessary layer of our software stack.
> RPC Support for QoS
> -------------------
>
> Key: HADOOP-9194
> URL: https://issues.apache.org/jira/browse/HADOOP-9194
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 2.0.2-alpha
> Reporter: Luke Lu
>
> One of the next frontiers of Hadoop performance is QoS (Quality of Service).
> We need QoS support to fight the inevitable "buffer bloat" (including various
> queues, which are probably necessary for throughput) in our software stack.
> This is important for mixed workload with different latency and throughput
> requirements (e.g. OLTP vs OLAP, batch and even compaction I/O) against the
> same DFS.
> Any potential bottleneck will need to be managed by QoS mechanisms, starting
> with RPC.
> How about adding a one byte DS (differentiated services) field (a la the
> 6-bit DS field in IP header) in the RPC header to facilitate the QoS
> mechanisms (in separate JIRAs)? The byte at a fixed offset (how about 0?) of
> the header is helpful for implementing high performance QoS mechanisms in
> switches (software or hardware) and servers with minimum decoding effort.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
