[
https://issues.apache.org/jira/browse/YARN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613804#comment-14613804
]
Nam H. Do commented on YARN-2681:
---------------------------------
+ The release audit and findbugs warning are incorrectly reported due to the
compile of trunk hadoop-mapreduce-client-app was broken in the pre-patch phase.
In fact, hs_err_pid3335.log is generated during this compile (see
trunkFindBugsOutputhadoop-mapreduce-client-app.txt), and as it is broken, it
wasn't able to detect the related findbugs warning (this warning belongs to
NodeLabelExpression feature!).
+ The failed tests are not related to the patch. I verified that all these
tests (after merged with the latest trunk) are passed in our local environment
(Java 1.7.0_80 HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)/Linux
xcp116 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux)
> Support bandwidth enforcement for containers while reading from HDFS
> --------------------------------------------------------------------
>
> Key: YARN-2681
> URL: https://issues.apache.org/jira/browse/YARN-2681
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
> Affects Versions: 2.5.1
> Environment: Linux
> Reporter: Nam H. Do
> Labels: BB2015-05-TBR
> Fix For: 2.7.0
>
> Attachments: Traffic Control Design.png, YARN-2681.001.patch,
> YARN-2681.002.patch, YARN-2681.patch
>
>
> To read/write data from HDFS on data node, applications establise TCP/IP
> connections with the datanode. The HDFS read can be controled by setting
> Linux Traffic Control (TC) subsystem on the data node to make filters on
> appropriate connections.
> The current cgroups net_cls concept can not be applied on the node where the
> container is launched, netheir on data node since:
> - TC hanldes outgoing bandwidth only, so it can be set on container node
> (HDFS read = incoming data for the container)
> - Since HDFS data node is handled by only one process, it is not possible
> to use net_cls to separate connections from different containers to the
> datanode.
> Tasks:
> 1) Extend Resource model to define bandwidth enforcement rate
> 2) Monitor TCP/IP connection estabilised by container handling process and
> its child processes
> 3) Set Linux Traffic Control rules on data node base on address:port pairs in
> order to enforce bandwidth of outgoing data
> Concept: http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
> Implementation:
> http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl.pdf
> http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl_UML_diagram.png
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
