[
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477813#comment-16477813
]
Bibin A Chundatt commented on YARN-8103:
----------------------------------------
Hi All,
Attaching first patch for CLI
Following main items i have covered in this patch
# Admin command and Client command separation
# Addition of command list attributes (Over all cluster attributes)
# Addition of command list *-nodetoattributes* and *-nodetoattributes -nodes
<hostname>*
# Addition of command *-attributestonodes* and *-attributestonodes -attributes
<>* (to filter by attributes)
# Nodes CLI command updation for adding attributes to nodes status
# Cluster CLI command updation for listing cluster attributes
# CLI node-attributes pattern seems to be not supported by script . Changed to
nodeattributes
{code}
Command error
root@bibinpc:/opt/apacheprojects/hadoop/YARN3409/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin#
./yarn node-attributes
/opt/apacheprojects/hadoop/YARN3409/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh:
line 2371: YARN_NODE-ATTRIBUTES_USER: bad substitution
/opt/apacheprojects/hadoop/YARN3409/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh:
line 2336: YARN_NODE-ATTRIBUTES_USER: bad substitution
/opt/apacheprojects/hadoop/YARN3409/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh:
line 2426: HADOOP_NODE-ATTRIBUTES_OPTS: bad substitution
Usage: yarn node-attributes
{code}
Bugs handled:
# {{AdminService}} IO exception need to be wrapped in RemoteException during
validation at server side. ()
{code}
root@bibinpc:/opt/apacheprojects/hadoop/hadoop-3.2.0-SNAPSHOT/bin# ./yarn
nodeattributes -replace "x:key(string)=value,key2=val2" -failOnUnknownNodes
2018-05-15 22:29:04,170 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8033
2018-05-15 22:29:04,381 INFO retry.RetryInvocationHandler: java.io.IOException:
Following nodes does not exist : [x, x]
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.validateAndFetch(AdminService.java:1047)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.mapAttributesToNodes(AdminService.java:987)
at
org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceManagerAdministrationProtocolPBServiceImpl.mapAttributesToNodes(ResourceManagerAdministrationProtocolPBServiceImpl.java:354)
at
org.apache.hadoop.yarn.proto.ResourceManagerAdministrationProtocol$ResourceManagerAdministrationProtocolService$2.callBlockingMethod(ResourceManagerAdministrationProtocol.java:315)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
{code}
# RMNode getAllAttributes to read from attributeManager . When attribute set
from CLI RMNode attributes was returning empty.
Sample outputs
# NodeAttributeCLI
{code}
root@bibinpc:/opt/apacheprojects/hadoop/hadoop-3.2.0-SNAPSHOT/bin# ./yarn
nodeattributes
Usage: yarn nodeattributes
Admin Commands:
-replace <"node1:attribute[(type)][=value],attribute1[=value],attribute2
node2:attribute2[=value],attribute3">
-add <"node1:attribute[(type)][=value],attribute1[=value],attribute2
node2:attribute2[=value],attribute3">
-remove <"node1:attribute,attribute1 node2:attribute2">
-failOnUnknownNodes
Client Commands:
-list
-nodestoattributes
-nodes <Host Names>
-attributestonodes
-attributes <Attributes>
-help [cmd] List help of commands
{code}
# NodeReport
{code}
Node Report :
Node-Id : bibinpc:35797
Rack : /default-rack
Node-State : RUNNING
Node-Http-Address : bibinpc:8042
Last-Health-Update : Wed 16/May/18 11:17:18:208IST
Health-Report :
Containers : 0
Memory-Used : 0MB
Memory-Capacity : 8192MB
CPU-Used : 0 vcores
CPU-Capacity : 8 vcores
Node-Labels :
Node Attributes :rm.yarn.io/GPU(STRING)=ARM
Resource Utilization by Node : PMem:10357 MB, VMem:10357 MB,
VCores:0.08330557
Resource Utilization by Containers : PMem:0 MB, VMem:0 MB, VCores:0.0
{code}
# NodesToAttributes
{code}
2018-05-16 23:19:33,919 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
Attribute Type
Value
localhost:
rm.yarn.io/GPU STRING
bibin
rm.yarn.io/apache-common-jar STRING
bibin
node3:
rm.yarn.io/apache-common-jar STRING
bibin
bibinpc:
rm.yarn.io/GPU STRING
ARM
x:
rm.yarn.io/key STRING
value
rm.yarn.io/key2 STRING
val2
node1:
rm.yarn.io/apache-ceqweqweommos STRING
bibiqenqweqweqwewewqe
rm.yarn.io/apache-commos STRING
bibinqwewewqe
rm.yarn.io/apache-ceqweommos STRING
bibiqenqwewewqe
rm.yarn.io/apache-common-jar STRING
bibin
{code}
# List Attributes
{code}
root@bibinpc:/opt/apacheprojects/hadoop/hadoop-3.2.0-SNAPSHOT/bin# ./yarn
nodeattributes -list
2018-05-16 23:22:27,420 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
Attribute Type
rm.yarn.io/GPU STRING
rm.yarn.io/key STRING
rm.yarn.io/key2 STRING
rm.yarn.io/apache-ceqweqweommos STRING
rm.yarn.io/apache-commos STRING
rm.yarn.io/apache-ceqweommos STRING
rm.yarn.io/apache-common-jar STRING
{code}
# Attributes to Node
{code}
root@bibinpc:/opt/apacheprojects/hadoop/hadoop-3.2.0-SNAPSHOT/bin# ./yarn
nodeattributes -attributestonodes
2018-05-16 23:23:33,034 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
Hostname Attribute-value
rm.yarn.io/GPU :
localhost bibin
bibinpc ARM
rm.yarn.io/key :
x value
rm.yarn.io/key2 :
x val2
rm.yarn.io/apache-ceqweqweommos :
node1 bibiqenqweqweqwewewqe
rm.yarn.io/apache-commos :
node1 bibinqwewewqe
rm.yarn.io/apache-ceqweommos :
node1 bibiqenqwewewqe
rm.yarn.io/apache-common-jar :
localhost bibin
node3 bibin
node1 bibin
{code}
> Add CLI interface to query node attributes
> -------------------------------------------
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Bibin A Chundatt
> Assignee: Bibin A Chundatt
> Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch,
> YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface
> for querying node attributes for each nodes and list all attributes in
> cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
