[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388420#comment-14388420 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #149 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/149/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/CHANGES.txt Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388440#comment-14388440 ] Hudson commented on YARN-2495: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #883 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/883/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388634#comment-14388634 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #140 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/140/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388611#comment-14388611 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2081 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2081/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389155#comment-14389155 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2099 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2099/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389086#comment-14389086 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #149 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/149/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387240#comment-14387240 ] Hudson commented on YARN-2495: -- FAILURE: Integrated in Hadoop-trunk-Commit #7467 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7467/]) YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) (wangda: rev 2a945d24f7de1a7ae6e7bd6636188ce3b55c7f52) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384972#comment-14384972 ] Wangda Tan commented on YARN-2495: -- Last call for comments, I'm planning to commit this by the end of this weekend. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382299#comment-14382299 ] Jian Fang commented on YARN-2495: - In a cloud environment such as Amazon EMR, a hadoop cluster is launched as a service by a single command line. There is no admin at all and everything is automated. The lables are basically of two types, one is static. For example, the nature of an EC2 instance such as spot or on-demand. The other is dynamic. For example, the cluster controller process can set an instance to be a candidate to be terminated in the case of graceful shrink so that resource manager will not assign new tasks to it. Most likely, the labels specified from each NM are static and are provided by a cluster controller process to write into yarn-site.xml based on EC2 metadata available on each EC2 instance. As a result, at least you should defined a static lablel provider (plus a dynamic lable provider? not sure) so that these lables are only sent to resource manager at NM registeration time. There is no point to add the static lables to each heartbeat. I think the idea of central and distributed label configurations are not ideal to use in a cloud environment. Usually we have a mix of static lables from each node and dynamic labels that are specified against the resource manager directly. Static and dynamic lable concepts are more appopriate at least for Amazon EMR. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382506#comment-14382506 ] Allen Wittenauer commented on YARN-2495: That's effectively what the executable interface is for Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382555#comment-14382555 ] Jian Fang commented on YARN-2495: - Great, thanks. Will try them. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382389#comment-14382389 ] Wangda Tan commented on YARN-2495: -- Hi [~john.jian.fang], Thanks for your comments, I'm not sure if I completely understood what you said. Did you mean there're two different types of NMs, which is: some labels are not changed in NM's lifetime, some labels could be modified when NM's running (I think the decommission case you provided is better to be resolved by graceful NM decommission instead of node label.). Having a centralized node label list is mostly for resource planning, you can take a look at conversions on YARN-3214 for more details about resource planning stuffs. Regardless of the centralized node label list in RM side, I think current implementation of attached patch should work for you. Even if labels could be modified via heartbeat, but you can simply not change them in your own script, if there's no changes of NM's labels, no duplicated data will be sent to RM side. Wangda Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382388#comment-14382388 ] Naganarasimha G R commented on YARN-2495: - Hi [~john.jian.fang], Well this jira is followed by YARN-2729, where in labels got from the script are passed as part of heartbeat which makes the distributed label configuration as dynamic. Also as part of this jira we have tried to ensure that only when there is change in labels we send and if not we do not send static lables to each heartbeat. And for your case if cluster controller process wants to label a node so that it can graceful shrink can be done in 2 ways: * Use REST API and change the label of the node to some unique label which is not visible to other users * After YARN-2729, may be you can have some script which has appropriate logic to update RM with some some unique label when it wants to shrink itself gracefully. Hope i have addressed your scenario Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382496#comment-14382496 ] Jian Fang commented on YARN-2495: - On each EC2 instance, the metadata about that instance such as its market type, i.e., spot or on-demand, CPUs, memory and etc are available when the instance starts up. All these information are injected to yarn-site.xml by our instance controller and they will not be changed afterwards. Different instances in an EMR cluster could have different static lables since one EMR hadoop consists of multiple instance groups, i.e., different types of instances. I think it is ok that no duplicated data are sent to RM if not NM lable changes. Thanks. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382501#comment-14382501 ] Jian Fang commented on YARN-2495: - BTW, I haven't gone through all the details of YARN-2492 yet, is it possible to provide a configuration to hook in different label providers on NM, for example, a third party one? (Sorry if this feature already exists). Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382542#comment-14382542 ] Wangda Tan commented on YARN-2495: -- bq. is it possible to provide a configuration to hook in different label providers on NM, for example, a third party one? (Sorry if this feature already exists). Yes, you can check in this patch, how LabelProvider created is leaving blank, and we have two JIRAs to make it configurable: - YARN-2729 for script based - YARN-2923 for config based This should be pluggable and new provider can be added in the future. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378203#comment-14378203 ] Wangda Tan commented on YARN-2495: -- Thanks for update. Patch LGTM, +1. will wait and commit in a few days if there's no opposite opinions. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377356#comment-14377356
]
Hadoop QA commented on YARN-2495:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706826/YARN-2495.20150324-1.patch
against trunk revision 9fae455.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7088//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7088//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376605#comment-14376605
]
Wangda Tan commented on YARN-2495:
--
Hmm.. {{StringArrayProto.stringElement - elements}} is still not changed in
latest patch, could you take a look again?
I meant to remove the string prefix, since the StringArrayProto already
indicates that. Beyond that, patch LGTM.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495.20150321-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377252#comment-14377252 ] Naganarasimha G R commented on YARN-2495: - Hi [~leftnoteasy], Oops mistake from my side, Have uploaded the patch with correction, Please check. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495.20150324-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372917#comment-14372917 ] Naganarasimha G R commented on YARN-2495: - Hi [~leftnoteasy] following test case failures are not related to the changes made in the patch Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495.20150321-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372280#comment-14372280 ] Wangda Tan commented on YARN-2495: -- 2) You decide. :) 5)/7) You're correct, provider should only know it's label instead of updated or not, so null from provider should be considered to be empty. And areNodeLabelsUpdated only receives non-null parameters, add a comment before areNodeLabelsUpdated to indicate null variables will be handled. Thanks, Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372524#comment-14372524
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706116/YARN-2495.20150321-1.patch
against trunk revision e1feb4e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7059//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7059//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495.20150321-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371830#comment-14371830
]
Naganarasimha G R commented on YARN-2495:
-
Thanks for review [~wangda] :
1) {{StringArrayProto.stringElement - elements}}, while naming had done based
on Protoc but as you mentioned seems like
elements would make more sense in terms of java api, will correct it as part of
the next patch.
2) {quote} After thought, I think optional bool areNodeLabelsAcceptedByRM = 7
\[default = false\]; should be true to be more defensive: We need make sure
there's no error when somebody forget to set this field. {quote}
Well my view as explained earlier is lil diff, as even if some body forget it
test case will ensure they do not miss it but the case which i mentioned
earlier check whether its legitamate scenario
??But consider the case where NM gets upgraded first then it should not be the
case NM sends labels and older RM ignores the additional labels but response by
default sends labels are accepted. And also felt by name/functionality, it
should be set to true only after RM accepts the labels??
3) will be taken care in next patch
4) {quote}NodeLabelsProviderService - NodeLabelsProvider, like most other
modules, we don't need to make service as a part of the classname, change sub
classes and NodeManager.createNodeLabelsProviderService as well.{quote}
Well i agree with this but long b4 you had only asked me to change it
NodeLabelsProviderService [Prev comment 4th
point|https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14181031page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14181031].
will get it done in next patch
5) {quote}NodeStatusUpdaterImpl.run:
617 int lastHeartbeatID = 0;
618 SetString nodeLabelsLastUpdatedToRM = null;
619 if (hasNodeLabelsProvider)
No matter if hasNodeLabelsProvider, nodeLabelsLastUpdatedToRM should be null?
By default is not change instead of empty, correct? {quote}
nodeLabelsLastUpdatedToRM = lastUpdatedNodeLabelsToRM i.e. its not the one
which will be sent as part of heartbeat (nodeLabelsForHeartbeat is used) and
nodeLabelsLastUpdatedToRM is reference for comparison whether labels have
changed since the last call. And as per the logic, NodeLabelsProvider even if
its return null we consider it as empty labels. So i feel its correctly set.
6) will take care in next patch
7) areNodeLabelsUpdated: Need check null? And could you add more test to cover
the case when new fetched node labels and/or last node labels are null?
{{Need check null}} has been taken care before calling this method, also you
had asked for commenting this which i have done in NodeLabelsProvider, wil do
this here too.
{{add more test to cover the case when new fetched node labels}} already there
in TestNodeStatusUpdaterForLabels.testNodeStatusUpdaterForNodeLabels(ln num 271
-277)
As explained in prev comment even though the NodeLabelsProvider return null
node labels we consider it as empty set hence there is no way last node labels
are null. Null is sent as part of heartbeat only when lastUpdatedNodeLabelsToRM
== nodeLabelsForHeartbeat . Will put some comments for this.
Please let me know your comments will try to give the patch ASAP.. :)
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370333#comment-14370333
]
Wangda Tan commented on YARN-2495:
--
Go through patch again, hopefully this is last round from my side :)
1) StringArrayProto.stringElement - elements
2) After thought, I think {{optional bool areNodeLabelsAcceptedByRM = 7
\[default = false\];}} should be true to be more defensive: We need make sure
there's no error when somebody forget to set this field.
3) testNodeHeartbeatRequestPBImplWithNullLabels: remove
{{original.setNodeLabels(null);}}, test should still pass.
4) NodeLabelsProviderService - NodeLabelsProvider, like most other modules, we
don't need to make service as a part of the classname, change sub classes and
NodeManager.createNodeLabelsProviderService as well.
5) NodeStatusUpdaterImpl.run:
{code}
617 int lastHeartbeatID = 0;
618 SetString nodeLabelsLastUpdatedToRM = null;
619 if (hasNodeLabelsProvider) {
{code}
No matter if hasNodeLabelsProvider, nodeLabelsLastUpdatedToRM should be null?
By default is not change instead of empty, correct?
6) nodeLabelsLastUpdatedToRM - lastUpdatedNodeLabelsToRM
7) areNodeLabelsUpdated: Need check null? And could you add more test to cover
the case when new fetched node labels and/or last node labels are null?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370386#comment-14370386
]
Hadoop QA commented on YARN-2495:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705721/YARN-2495.20150320-1.patch
against trunk revision 91baca1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7028//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7028//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495.20150320-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369909#comment-14369909
]
Wangda Tan commented on YARN-2495:
--
Hi [~Naganarasimha],
For 1)
I suggest to use a separated PB message, such as NodeLabelsProto (or make it
generic like StringArrayProto), which contains {{repeat string}}.
When using NodeIdToLabelsProto, but don't use nodeId, that will confuse people.
About test cases
1. NodeStatusUpdaterTest:
Some places need to cover:
- NM register, should check RTS (ResourceTrackerService) labels (done)
- NM heartheat, should check RTS labels (TODO)
- NM headtbeat without update, should check RTS received labels (TODO)
- NM heartbeat with update, should check RTS received labels (TODO)
2. TestResourceTrackerService
Test generally very good to me,
- lblsMgr - nodeLabelsMgr or labelsMgr
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370056#comment-14370056
]
Wangda Tan commented on YARN-2495:
--
ResourceTrackerForLabels.labels - lastReceivedLabels
{code}
262 // heartbeat without updating labels
263 nm.getNodeStatusUpdater().sendOutofBandHeartBeat();
264 resourceTracker.waitTillHeartBeat();
265 resourceTracker.resetNMHeartbeatReceiveFlag();
{code}
Need add some check for this operation?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366562#comment-14366562
]
Naganarasimha G R commented on YARN-2495:
-
1. Well i think i am missing something or you have miss-read my patch.
* in both request Protos i have used NodeIdToLabelsProto {{optional
NodeIdToLabelsProto nodeLabels = 8;}} and {{optional NodeIdToLabelsProto
nodeLabels = 4;}} and not used {{repeat string nodeLabels}} directly
* Test cases TestYarnServerApiClasses.testNodeHeartbeatRequestPBImpl,
testNodeHeartbeatRequestPBImplWithNullLabels,
testRegisterNodeManagerRequestWithNullLabels and
testRegisterNodeManagerRequestWithValidLabels validates that the approach in
the patch supports null and filled labelsset, manually tested for empty set and
as expected that too worked. will get this test case added in next patch
* thought of creating a new proto like {{StringSetProto}} but felt why create
another proto class just for this purpose and you too had mentioned to use
{{NodeIdToLabelsProto}} hence made use of existing proto class
2. {{Typo, lable - label,}} : oops because of typo in proto, generated methods
also had issues hence proto and places accessing these methods(6 instances)
have this error. Will get it corrected in next patch.
3. ??optional bool areNodeLablesAcceptedByRM = 7 \[default = false\], I think
default should be true.??
Personally i felt it should not matter as I am explicitly handling in the code.
But consider the case where NM gets upgraded first then it should not be the
case NM sends labels and older RM ignores the additional labels but response by
default sends labels are accepted. And also felt by name/functionality, it
should be set to true only after RM accepts the labels
4,5,6
--will get it corrected as part of next patch.
Also one favor, it would be helpful if you review the test-cases and give feed
back on them too, as it will reduce my effort in creating multiple patches. I
understand that its huge size patch but anyway i feel major
aspects/functionality seems to be stable with the last patch.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366312#comment-14366312
]
Wangda Tan commented on YARN-2495:
--
Thanks for updating.
1. I saw you're trying to make {{repeat string nodeLabels}} to be 3 different
values: null, empty, and non-empty. But I'm not sure if this works in PB, could
you write a test to verify that? (Not-set/Set-nothing/Set-value to
RegisterNodeManagerRequestPBImpl.nodeLabels, create a new
RegisterNodeManagerRequestPBImpl from old.getProto(), and get nodeLabels() to
see if it works).
If this doesn't work, you can create a PB message like StringSetProto use it in
messages like RegisterNodeManagerRequest, which can support null/empty/non-empty
2. Typo, lable - label, I found several in your patch.
3. optional bool areNodeLablesAcceptedByRM = 7 \[default = false\], I think
default should be true.
4. NodeStatusUpdaterImpl: no need to call nodeLabelsProvider.getNodeLabels()
twice when register/heartbeat
5. HeartBeat - Heartbeat
6. NodeStatusUpdaterImpl: When labels are rejected by RM, you should log it
with diag message.
Will include test review in next round.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366236#comment-14366236
]
Hadoop QA commented on YARN-2495:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705179/YARN-2495.20150318-1.patch
against trunk revision 968425e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7003//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7003//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495.20150318-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357928#comment-14357928
]
Naganarasimha G R commented on YARN-2495:
-
bq. It's just that I don't like this areNodeLabelsSetInReq flag in the
protocol. Are there other ways of achieving this?
Other way out is always send the set of labels as part of every heartbeat. We
wanted to avoid this traffic, hence initially we came up with this approach
when we were supporting multiple labels for a node (may be in future we might
support multiple labels again right ?)
{{I think treating invalid labels as a disaster case will be, well, a
disaster.}} : liked the sentence :)
bq. How about we let the node run (just like we let an unhealthy node run) and
report it in the diagnostics? I'm okay keeping that same behavior during
registration too.
Yes this is similar to the earlier behavior we had in the Dec's patch ,
Additionally to inform back the failure, we had added one flag to inform NM
whether RM accepted the labels or not and Diag Message was also set with
appropriate message. Whether this approach is fine ?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357852#comment-14357852 ] Vinod Kumar Vavilapalli commented on YARN-2495: --- bq. Well as craig informed, RegisterNodeManagerRequestProto.nodeLabels is already a set but as by default empty set is provided by protoc, its req to inform whether labels are set as part of request hence areNodeLabelsSetInReq is required. It's just that I don't like this _areNodeLabelsSetInReq_ flag in the protocol. Are there other ways of achieving this? bq. Well i am little confused here, As per wangda's earlier comment i understand that it was your comment to send shutdown (which i felt correct in terms of maintenance) I think treating invalid labels as a disaster case will be, well, a disaster. How about we let the node run (just like we let an unhealthy node run) and report it in the diagnostics? I'm okay keeping that same behavior during registration too. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358109#comment-14358109
]
Wangda Tan commented on YARN-2495:
--
Hi [~Naganarasimha],
For areNodeLabelsSetInReq, one appoarch to avoid it is, you can add a PB
structure (like NodeLabelsProto, could placed in yarn_protos.proto, like
NodeIdToLabelsProto), which contains {{repeated string nodeLabels}}. It will be
a part of RegisterProto and HeartbeatProto, and it could be null. Null means
not set, agree?
Regarding the invalid node label issue of registration/hearbeating, bring down
the whole cluster when label is invalid is a serious problem as mentioned by
Craig, maybe you can:
If label(s) of a node being reporting is invalid, we can
- Show/log diagnostic in RM (nodes) page and NM page, saying label is invalid.
(Need modify web UI, can be done in a separated task)
- Make the node's labels to be empty, so that applications can continue use it.
Thanks,
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355283#comment-14355283 ] Naganarasimha G R commented on YARN-2495: - Hi [~wangda] [~vinodkv], Also one query/suggestion, would it be better to have single configuration for node labels yarn.node-labels.configuration.type=centralized/distributed/disabled (with default as disabled) instead of currently available 2 configurations i.e. yarn.node-labels.enabled and yarn.node-labels.configuration.type, so that we can avoid one more configuration ? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355437#comment-14355437 ] Wangda Tan commented on YARN-2495: -- Actually I've thought about this before, but since the node-labels.enabled is already shipped with Hadoop 2.7, we cannot change this. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355436#comment-14355436
]
Wangda Tan commented on YARN-2495:
--
Hi Naga,
--
bq. Well modifications side is clear to me but is it good to allow the
configurations being different from NM and RM ? Infact i wanted to discuss
regarding whether to send shutdown during register if NM is configured
differently from RM, but waited for the base changes to go in before discussing
new stuff.
It is not make the configuration different, my thinking of this is, NM doesn't
need understand what is distribtued-configuration, admin should know it. When
the node-label is distributed-configuration, NM should go ahead and properly
configure script provider, etc.
So we aren't trying to create different, we just eliminate one option in NM
side.
--
bq. I feel better to Log this as Error as we are sending the labels only in
case of any change and there has to be some way to identify if labels for a
given NM and also currently we are sending out shutdown signal too.
What I meant are the two line,
{code}
498 LOG.info(Node Labels { + StringUtils.join(,, nodeLabels)
499 + } from Node + nodeId + were Accepted from RM);
{code}
I guess you may misread my comment
--
For the field to indicate if node labels are set in
NodeHeartbeatRequest/NodeRegistrationRequest, there're two proposals:
- setAreNodeLabelsSetInReq
- setAreNodeLabelsUpdated
Which one you prefer, Vinod/Craig? I vote the latter one :)
--
bq. We should not even accept a node's registration when it reports invalid
labels
IIUC, now the patch already reject node when it reports invalid labels.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355448#comment-14355448 ] Vinod Kumar Vavilapalli commented on YARN-2495: --- bq. I'm wondering if we might introduce a situation where a script error or other configuration issue could bring down an entire cluster Exactly my concern too. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353241#comment-14353241 ] Naganarasimha G R commented on YARN-2495: - Most of the find bugs reported are from Fair scheduler and nothing to do with changes in the patch and tests failed are due to timeout and those are also not related to the patch. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353217#comment-14353217
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703435/YARN-2495.20150309-1.patch
against trunk revision 5578e22.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
org.apache.hadoop.yarn.server.resourcemanager.TestRM
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6894//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6894//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6894//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353353#comment-14353353
]
Wangda Tan commented on YARN-2495:
--
For your comments:
1) For the name, do you think is setAreNodeLabelsUpdated a better name since it
avoids set occured twice :) (I understand this needs lots of refactorings, if
you have any suggestions, we can finalize it before renaming.
5) I made a mistake that sent an incompleted comment :-p, what I wanted to say
is:
It will be problematic to ask admins make NM/RM configuration keep
synchronized, so I don't want (and also not necessary) NM depends on RM's
configuration.
So I suggest to make a changes:
- In NodeManager.java: when user doesn't configure provider, it should be null.
In your patch, you can return a null directly, and YARN-2729 will implement the
logic of instancing provider from config.
- In NodeStatusUpdaterImpl: avoid using {{isDistributedNodeLabelsConf}}, since
we will not have distributedNodeLabelConf in NM side if you agree on
previously comment, instead, it will check null of provider.
Regarding your fail-fast concern, it shouldn't be a problem if you agree on
comment I just made. (I know there could be some back-and-forth comment from my
side on this, I feel sorry about this since this feature is evolving itself,
please just feel free to let me know your ideas.).
7) I should address your question on 5).
8) You can add an additional comments in line 626 for this.
9) Took a look at TestNodeStatusUpdater, your comment make sense to me, it's a
very complex class, you can just leave this comment alone.
10) Few comments for your added code:
- updateNodeLabelsInNodeLabelsManager - updateNodeLabelsFromNMReport
- {{LOG.info(... accepted from RM}}, use LOG.debug and check {{isDebugEnabled}}.
- Make errorMessage clear: indicate 1# this is node labels reported from NM,
and 2# it's failed to be put to RM instead of not properly configured.
In addition:
Another thing we should do is, when distributed node label configuration is
set, any direct modify node to labels mapping from RMAdminCLI should be
rejected (like -replaceNodeToLabels). This can be done in a separated JIRA.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353639#comment-14353639 ] Vinod Kumar Vavilapalli commented on YARN-2495: --- Ah, right. Forgot about that. Given that, it seems that we have the following # Node reports with invalid labels during registration - we reject it rightaway # Node gets successfully registered, but then the labels script starts generating invalid labels mid way through I think in case (2), we are better off ignoring the newly reported invalid labels, report this in the UI/NodeReport and let the node continue running. Thoughts? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353397#comment-14353397 ] Craig Welch commented on YARN-2495: --- -re bq. How about we simply things? Instead of accepting labels on both registration and heartbeat, why not restrict it to be just during registration? As I understand the requirements, it's necessary to handle the case where the derived set of labels changes during the lifetime of the nodemanager, e.g. externally libraries might be installed or some other condition may change which effects the labels no nodemanager re-registration is involved, and yet the changed labels need to be reflected Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353383#comment-14353383 ] Vinod Kumar Vavilapalli commented on YARN-2495: --- Quick comments - configuration.type - configuration-type - Should RegisterNodeManagerRequestProto.nodeLabels be a set instead? - Do we really need NodeHeartbeatRequest.areNodeLabelsSetInReq()? Why not just look at the set as mentioned in the previous comment? - RegisterNodeManagerRequest is getting changed. It will be interesting to reason about rolling-upgrades in this scenario. - How about we simply things? Instead of accepting labels on both registration and heartbeat, why not restrict it to be just during registration? - We should not even accept a node's registration when it reports invalid labels? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353897#comment-14353897 ] Wangda Tan commented on YARN-2495: -- I think the two issues are identical, and we should have a consistent way to handle them. If we stop node when any invalid labels during registration, we should stop node when same issue happened when heartbeat after registration. I think we can either allow them running or stop both of them, I'm fine with both approach. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495.20150305-1.patch, YARN-2495.20150309-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354362#comment-14354362
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda],
1) IMO method name was not readable when it was {{setAreNodeLabelsSet}} but i
have changed it to {{setAreNodeLabelsSetInReq}} i feel this is sufficient.
setAreNodeLabelsUpdated is same as earlier for which Craig had commented (which
i also feel valid)
{quote}
I would go with areNodeLablesSet (all isNodeLabels = areNodeLabels
wherever it appears, actually) - wrt Set vs Updated - this is primarily a
workaround for the null/empty ambiguity and I think this name better reflects
what is really going on (am I sending a value to act on or not), but I also
think that this is a better contract, the receiver (rm) shouldn't really care
about the logic the nm side is using to decide whether or not to set it's
labels (freshness, updatedness, whatever), so all that should be communicated
in the api is whether or not the value is set, not whether it's an
update/whether it's checking freshness, etc. that's a nit, but I think it's a
clearer name.
{quote}
Yes true lets finalize the name this time after that will start working on the
patch if not it will be a wasted effort
5)
{quote}
It will be problematic to ask admins make NM/RM configuration keep
synchronized, so I don't want (and also not necessary) NM depends on RM's
configuration.
So I suggest to make a changes: In NodeManager.java: when user doesn't
configure provider, it should be null. In your patch, you can return a null
directly, and YARN-2729 will implement the logic of instancing provider from
config. In NodeStatusUpdaterImpl: avoid using isDistributedNodeLabelsConf,
since we will not have distributedNodeLabelConf in NM side if you agree on
previously comment, instead, it will check null of provider.
{quote}
Well modifications side is clear to me but is it good to allow the
configurations being different from NM and RM ? Infact i wanted to discuss
regarding whether to send shutdown during register if NM is configured
differently from RM, but waited for the base changes to go in before discussing
new stuff.
8) ??You can add an additional comments in line 626 for this.?? Ok will add a
comment in LabelProvider.getLabels , Idea is LabelProvider is expected to give
same Labels continiously untill there is a change and if null or empty is
returned then No label is assumed
10) {{updateNodeLabelsInNodeLabelsManager - updateNodeLabelsFromNMReport}} :
will take care in next patch
{{LOG.info(... accepted from RM, use LOG.debug and check isDebugEnabled.}} : I
feel better to Log this as Error as we are sending the labels only in case of
any change and there has to be some way to identify if labels for a given NM
and also currently we are sending out shutdown signal too.
??Make errorMessage clear: indicate 1# this is node labels reported from NM,
and 2# it's failed to be put to RM instead of not properly configured.??
i think i have captured first point, but any way will reframe it as {{Node
Labels labels reported from the NM with id nodeID were rejected from RM
with exception message as exceptionMsg.}}
??Another thing we should do is, when distributed node label configuration is
set, any direct modify node to labels mapping from RMAdminCLI should be
rejected (like -replaceNodeToLabels).?? Will work on this once 2495 and 2729
are done ..
Thanks [~vinodkv] [~cwelch] for reviewing it
??configuration.type - configuration-type?? will take care in next patch
{quote}
Should RegisterNodeManagerRequestProto.nodeLabels be a set instead?
Do we really need NodeHeartbeatRequest.areNodeLabelsSetInReq()? Why not just
look at the set as mentioned in the previous comment?
{quote}
Well as craig informed, RegisterNodeManagerRequestProto.nodeLabels is already
a set but as by default empty set is provided by protoc, its req to inform
whether labels are set as part of request hence areNodeLabelsSetInReq is
required.
??RegisterNodeManagerRequest is getting changed. It will be interesting to
reason about rolling-upgrades in this scenario.??
Well though i am not much aware of Rolling upgrades, i don't see any problems
in a normal case because RM tries to read the labels from NM's req only when
its distributed conf and also {{areNodeLabelsSetInReq}} is by default false.
But I had queries when some existing setup they want to modify to distributed
conf setup
# Whether we need to send shutdown during register if NM is configured
differently from RM ?
# Will the new configurations be added in NM and RM and then Rolling upgrade
will be done ? or we do rolling upgrade first and then reconfigure restart
RM's and NM's
??How about we simply things? Instead of accepting labels on both registration
and heartbeat, why not restrict it to be just during registration??
Well i have
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349336#comment-14349336
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702856/YARN-2495.20150305-1.patch
against trunk revision 28b85a2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6864//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6864//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6864//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349708#comment-14349708
]
Wangda Tan commented on YARN-2495:
--
1) Changes of CommonNodeLalelsManager is uncessary
2) NodeHeartbeat's protobuf field name should consistent with java class method
name:
nodeLabels- areNodeLabelsSet
3) NodeHeartbeatRequestPBImpl: getter/setter should consistent, like
get/setAreNodeLabelsSet
4) Same comment on RegisterNodeManagerRequestPBImpl
5) Integration with NM (NodeManager.java):
I think we may not need to check centralized/distributed configuration here,
centralized/distributed is a config in RM side. In NM side, it should be how to
get node labels, if user doesn't configure any script file for it, it should be
null and no instance of NodeLabelProviderService will be added to NM.
So back to code, you can just leave getNodeLabelsProviderService(..), which
will be implemented in YARN-2729.
If you agree, we need change the name {{isDistributedNodeLabelsConf}} to
6) Why change this to notifyAll? I think it may be unnecessary:
{code}
synchronized (this.heartbeatMonitor) {
this.heartbeatMonitor.notify();
}
{code}
7) isDistributedNodeLabels seems not so necessary here, and if you agree with
5), it's better to remove the field.
8) Add null check or comment (provider returned node labels will always be
not-null, for areNodeLabelsUpdated in NodeStatusUpdaterImpl
9) Since we already have TestNodeStatusUpdater, it's better to merge
TestNodeStatusUpdaterForLabels to it.
10) ResourceTrackerService:
When register and found illegal labels, this should happen before instanting
RMNodeImpl
Maybe you can still merge some common code for register/update?
Will include more tests reviewing in next turn. And could you take a look at
failed tests/findbugs, etc.?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495.20150305-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329376#comment-14329376
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda],
Thanks for discussing and summarizing this, few points to discuss:
# YARN-2923 (configuration based NodeLabelProviderService), what is the scope
of this jira as its again one more classification under {{distributed}} ?
whether is it req?
# Apart from the Vinod's suggestion, i would like to add one more change : In
some jiras we were discussing to have fail fast approach i.e. in our case if we
configure script based distributed configurations and the script file doesn't
exist, rights not sufficient etc.. then NM should fail to start.
# yarn-2980 is mostly just moving NodeHealthScriptRunner to HadoopCommon and
the code in it differs a lot than what we req, if you remember earlier i had
refactored that code and make it reusable for our scenario but you had
suggested not to touch the existing NodeHealthScriptRunner and to make
ScriptBasedNodeLabelsProvider similar NodeHealthScript serving the needs of
Node labels.
# Earlier target version was mentioned as 2.6, do we modify it to 2.7 or 2.8?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329399#comment-14329399 ] Wangda Tan commented on YARN-2495: -- For your questions: 1. It's not first priority, let's just keep it here and move on with the script one 2. That's make sense, and could you take a look at the health check script, is it as same as you mentioned? (I guess so) 3. I can remember that :), make sense to me. If there's any merge needed, we can tackle it in a separated JIRA. 4. I just marked to 2.8, since 2.7 is planned to be released within 2 weeks, I think it will be hard to fit in that timeframe. IIRC, this JIRA contains a abstract node label provider only. I suggest don't add more options for configuring node label provider for now (saying, set provider=script-based-provider), to avoid potentially unnecessary switches. For now, as we discussed, you can leave an empty provider implementation (but need proper impl to do tests), script-based-provider can be addressed separately. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327716#comment-14327716 ] Craig Welch commented on YARN-2495: --- So, here's my proposal [~Naganarasimha] [~leftnoteasy], take a minute and consider whether or not DECENTRALIZED_CONFIGURATION_ENABLED is more likely to cause difficulty than prevent it, as I'm suggesting, and then you all can decide to keep it or not as you wish - I don't want to hold up the way forward over something which is, on the whole, a detail... Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327718#comment-14327718
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685787/YARN-2495.20141208-1.patch
against trunk revision 2fd02af.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6670//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328370#comment-14328370
]
Wangda Tan commented on YARN-2495:
--
[~cwelch],
Thanks for your thoughtful inputs, configuration switches include changes
proposed in this patch is:
{code}
node-label
/ \
enable disable
/ \
centralized distributed
| |
node-label-store node-manager script
need to be configured need to be configured in NM
{code}
You can take a look at discussion of why we do YARN-2800, we need
enable/disable this feature (not enable/disable centralized API).
And there're some difference between centralized/distributed configure
- We don't need to store node-label mappings since NM will store it locally
- We cannot mix them together as I mentioned in
https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14317048page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14317048.
So there're totally 3 options of node-label feature itself in RM's view:
\{disabled, centralized, distributed\}. I don't think we can eliminate any of
them, and I didn't see your suggestion can make it :).
So if you agree, I suggest we can move forward this with following changes:
- Change {{decentralized-configuration.enabled}} to {{input}}, accept value of
\{centralized, distributed\} (by default is centralized)
- Combining suggestion in
https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14317048page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14317048.
- Not sure if https://issues.apache.org/jira/browse/YARN-2980 will impact
implementation of this patch (I haven't looked at it), but it's best to reusage
existing code.
Sounds good? [~Naganarasimha] please let me know if you have any other ideas :).
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323076#comment-14323076 ] Craig Welch commented on YARN-2495: --- My point is that everything necessary to manage labels properly exists without DECENTRALIZED_CONFIGURATION_ENABLED, it is a duplication of existing functionality. The user controls this by: 1. choosing to specify or not specify a way of managing the nodes at the node manager 2. choosing to set or not set node labels and associations using the centralized apis ergo, DECENTRALIZED_CONFIGURATION_ENABLED is completely redundant, it provides no capabilities not already present. Users will need to understand how the feature works to use it effectively anyway, there is no value add by requiring that they repeat themselves (both by specifying a way of determining node labels at the node manager level and by having to set this switch.). My prediction is that, if the switch is present, it's chief function will be to confuse and annoy users when they setup a configuration for the node managers to generate node labels and then the labels don't appear in the cluster as they expect them to. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320549#comment-14320549 ] Naganarasimha G R commented on YARN-2495: - Hi [~wangda], Thanks for sharing your thoughts bq. mixing decentralized/centralized is dangerous and will cause non-determinate result. My opinion is also same, i feel it should not be mixed. And would like to resume working (rebase rework on the review comments) on this once all are in the sync with the core objective of this jira, So [~cwelch] can you please share your thoughts on this at the earliest. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317048#comment-14317048 ] Wangda Tan commented on YARN-2495: -- Hi [~cwelch], Thanks for jumping in and providing your thoughts, and really sorry for the late response. I think biggest concern of you is about DECENTRALIZED_CONFIGURATION_ENABLED, let me talk about my thinkings :) IMHO, mixing decentralized/centralized is dangerous and will cause non-determinated result. You may think about merging them together, such as some labels set by admin using RMAdminCLI, and some others are set by NM. But I can give you an example shows it is still non-determinated even if we have +/- for ResourceTracker protocol: - Assume a node has label x,y (reported +x,+y) - RMAdmin remove y from the node (-y) - NM failure then restart, and report it has x,y (+x, +y). What should labels on the node be? I also don't like adding too much switches in configuration, but it seems a good way that we can support both with determinated behavior. For your other suggestions, - Name changes is-are, - Make RegisterNodeManagerRequest consist wiht NodeHeartbeatRequest I all agree with One more suggestion (as per suggested by [~vinodkv]), when there's anything wrong with node label reported from NM, we should fail NM (ask it to shutdown and give it proper diagnostic message). This is because if NM report a label but rejected, even if RM tell NM this, NM cannot handle it properly except print some error messages (we don't have smart logic now). Which will lead to problems in debugging (A NM reported some label to RM but scheduler failed allocating containers on the NM). To avoid it, a simple way is to shutdown the NM and admin can take a look at what happened. Thoughts? Wangda Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294441#comment-14294441 ] Craig Welch commented on YARN-2495: --- [~Naganarasimha] I understand the desire to have the feature, it does seem more like a convenience / simplification measure than an introduction of something that can't be otherwise accomplished, but convenience and simplification can matter a great deal, so why not :-) bq. There will be always confusion ... Do we need to And or OR What I'm getting at is that I think there are just too many switches and knobs in play once you consider the flag DECENTRALIZED_CONFIGURATION_ENABLED in addition to the other configuration relationships (defining the configuration script to do the update from nodes), I think that the act of configuring something to send node labels from the node manager is sufficient intent that it is the desired behavior, and the additional DECENTRALIZED_CONFIGURATION_ENABLED is just an extra ceiling for someone to bump their head against while setting this up. wrt supporting add vs replace behavior, I think that as it's described now the idea is to just support replace form the node script, meaning that it will effectively be the only definition used when it is active (which is fine for many cases). In the future, if there is a need for hybrid configuration of labels that can become an enhancement. An option would be to use a different parameter for a script which will do add and remove instead of replace, and then say have it return + or - (for add and remove) with the label instead of a fixed set of labels for replacement. From what I see above, the replacement approach, where the script determines the full label set, looks to be the immediate need - the other could be added in a compatible way later if it was needed. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14246882#comment-14246882
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~cwelch]
Thanks for taking time and reviewing.
bq. This can already be accomplished using the admin cli from a script or
calling a web service from the node
IIUC the current way to accomplish running the script is through configuring
the label script in health monitor script ? please correct me if i am wrong.
If it is so i see following problems
* It seems more like a workaround solution(or more like break in the health
check functionality) to get the same thing acheived
* One important point as [~aw] mentioned developer needs to take care of
security issues which might be too much effort
* Feels like extra effort needs to be put by the developer to get it done which
takes care of security, error handling , retries etc ... apart from the actual
script which might be very simple
* Documentation and conveying the functionality to the users would be difficult
* What if admin wants both health script and node labels script ?
bq. Is this all just to support putting labels into the node-managers
configuration file and introducing them that way? Do we have a solid need for
that? It's not needed for the dynamic script case, which is all I've seen
discussed here from a requirements perspective (putting it into the config
file / adding it to the heartbeat is implementation, I don't see a requirement
for it as such).
As [~aw] mentioned for this, main idea is not to support through configuration
file but run the configured script periodically and to directly update the RM
when change in the labels. Also it can do the error handling as required if RM
doesn't accept it.
bq. but I don't see any reason to have conditional enablement of the node
manager side of things as well as a provider specification and I think it just
adds unnecessary complexity and possible surprise at configuration time
Well this topic is debatable, but few points which i feel to be addressed if we
want to support both central and distributed simultaneously :
# There will be always confusion which labels needs to be taken suppose user
updates some labels for a node and NM label script doesnt have them then which
do we give priority ?
# Do we need to And or OR when the labels are updated either by script /by
admin CLI / REST web service
AFAIU [~gp.leftnoteasy] wanted to do these in stages and in the first stage to
avoid confusion it would be better to have mutually exclusive i.e either
central or distributed will be supported and if its distributed CLI
modifications will be disabled (YARN-2728) and if central, labels from
heartbeat will be discarded.
+1 for changes which you mentioned in {{NodeHeartbeatRequest}}
{{RegisterNodeLabelManagerResponse}} but for {{RegisterNodeManagerRequest}} yes
it would be easy to read the interface if both are same but as explained in
previous point if either one of them(central/distributed) then in case of
distributed it anyway needs to send what ever it has and non distibuted case RM
doesnt care abt it. So there was no need for it.
It would be helpful if [~gp.leftnoteasy] [~vinodkv] can further provide
their views on the requirement level comments which has been raised by [~cwelch]
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243171#comment-14243171 ] Craig Welch commented on YARN-2495: --- Sorry if I'm jumping in late and asking redundant questions as a result, but I've gone through the various related jiras and the design documents (incl updates) ( and the patch :-) ) and I have some requirements related questions as a result. Just a bit of background to make sure I understand things - it appears that we've settled on two different but related features here: 1. To enable node labels to be added or removed for a given node without validation against a centralized list of node labels (not on this patch, but relevant to the discussion) and 2. To enable node managers to specify their node labels based on local configuration and scripting (this patch is specific to that feature). These are, strictly speaking, orthogonal, but may be used together and will provide something more in a 'combined feature' A couple things about this feature (2) - I don't believe that it is necessary to add the node label configuration to the local configuration (yarn-site) or the heartbeat as such to enable configuration of labels for a node from the node in a decentralized fashion (e.g. a script on the node saying put these labels on me). This can already be accomplished using the admin cli from a script or calling a web service from the node (most likely the former, but either is possible...), so I don't think we need this change to support the script case, it's already possible to write a script to add a label to a node on the fly today without any changes. To make this dynamic we would need feature 1, from the above, but that's not covered in this patch / is a separate discussion / and also does not require this change. Also, I don't see how this change allows a script to dynamically configure labels unless it was changing the yarn-site or the like (I may have missed it, but I don't see that logic here) - and in any case, it would not be necessary to add this logic to support that sort of configuration as I pointed out. Is this all just to support putting labels into the node-managers configuration file and introducing them that way? Do we have a solid need for that? It's not needed for the dynamic script case, which is all I've seen discussed here from a requirements perspective (putting it into the config file / adding it to the heartbeat is implementation, I don't see a requirement for it as such). In a nutshell - do we need change 2 (this), or do we really just need change 1 (eliminating validation of labels against a centralized list, at least as a configurable option)? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243197#comment-14243197 ] Craig Welch commented on YARN-2495: --- So, I see the language around the script based vs the conf based provider, etc, so I assume that's where the scripting side comes in. However, it's still not clear to me that it's really a good idea to add all of this when there is already a way to accomplish the activity with a script - and already ways to run scripts from the node manager... Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243217#comment-14243217 ] Allen Wittenauer commented on YARN-2495: bq. This can already be accomplished using the admin cli from a script or calling a web service from the node Think about the secure cluster case. A whole new level of complexity is required to get this functionality using your proposed method vs. having the NM just run the script itself. bq. Is this all just to support putting labels into the node-managers configuration file and introducing them that way? Do we have a solid need for that? No, this is so we *don't* have to have hard-coded labels in a file. If we are doing an external software change, we need to be able to reflect that change up the chain. Think rolling upgrade. Think multiple service owners. FWIW, yes, we do have a solid need for this feature. Almost every ops person I talked to has said they'd likely make use it for the exact use cases I've highlighted above. Being able to roll out new versions of Java and make scheduling decisions on its installation is *extremely* powerful and useful. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243283#comment-14243283 ] Craig Welch commented on YARN-2495: --- Other comments on the patch as such, assuming we really do need this part of the change... DECENTRALIZED_CONFIGURATION_ENABLED et all - I do see the basis for enabling and disabling the enforcement of a centralized list and discussion around that, but I don't see any reason to have conditional enablement of the node manager side of things as well as a provider specification and I think it just adds unnecessary complexity and possible surprise at configuration time - at the level of this configuration I think it should just be enabled (and I don't mean just by default, I mean if we add this, it should just be a way to manage node labels, not conditionally enabled or disabled, any more than the web service or cli are conditionally enabled or disabled, and so we don't have this parameter / it's associated branching at all). I think the default node labels provider service should definitely be a null provider that always returns an empty list and areLabelsUpdated false - this takes out the need to decide which is default, a no-op one is, and it allows us to get rid of the extra enabled/disabled configuration above without adding a new configuration (the provider will be specified anyway if the feature is going to be used) NodeHeartbeatRequest - isNodeLabelsUpdated - I would go with areNodeLablesSet (all isNodeLabels = areNodeLabels wherever it appears, actually) - wrt Set vs Updated - this is primarily a workaround for the null/empty ambiguity and I think this name better reflects what is really going on (am I sending a value to act on or not), but I also think that this is a better contract, the receiver (rm) shouldn't really care about the logic the nm side is using to decide whether or not to set it's labels (freshness, updatedness, whatever), so all that should be communicated in the api is whether or not the value is set, not whether it's an update/whether it's checking freshness, etc. that's a nit, but I think it's a clearer name. RegisterNodeLabelManagerResponse - get/set IsNodeLabelsAcceptedByRM - I would make it get/set AreNodeLablesAcceptedByRM (and on imples, etc, of course) RegisterNodeManagerRequest - missing spaces in args (l 42) also, assuming we drop the distributed on/off config as I'm suggesting, you'll need the areNodeLablesSet to be passed here as well. (I also like this better because it harmonizes the api between registration and heartbeat, which is easier to understand b/c they are doing the same thing / should do it the same way). Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14239592#comment-14239592 ] Naganarasimha G R commented on YARN-2495: - Hi [~wangda] All the test case issues reported seems to be not related to my changes . Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238157#comment-14238157 ] Wangda Tan commented on YARN-2495: -- [~Naganarasimha], I meant you can leave it empty in this patch and set it after you have completed script/conf-based patch, just add a TODO comment should be fine. Make sense? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238336#comment-14238336
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685787/YARN-2495.20141208-1.patch
against trunk revision 144da2e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 38 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6036//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6036//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6036//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6036//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495.20141208-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235317#comment-14235317
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda],
Yes i wanted to completely remove it, but more than test cases using it was
used in NodeManager.createNodeLabelsProviderService()
{quote}
Class? extends NodeLabelsProviderService labelsProviderClass =
conf.getClass(YarnConfiguration.NM_NODE_LABELS_PROVIDER_CLASS,
*ConfigurationNodeLabelsProvider.class*,
NodeLabelsProviderService.class);
provider = labelsProviderClass.newInstance();
{quote}
Which i felt is required and hence gave a stub class to segregate the patch
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235363#comment-14235363
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685133/YARN-2495.20141204-1.patch
against trunk revision 0653918.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6001//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6001//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495.20141204-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or
using script suggested by [~aw] (YARN-2729) )
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235891#comment-14235891 ] Wangda Tan commented on YARN-2495: -- [~Naganarasimha], You don't have to specify the default implementation here. I think you can set the default implementation class to be null, with this review completed, we can decide either script-based or conf-based provider as the default implementation. Make sense? Thanks, Wangda Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234506#comment-14234506 ] Wangda Tan commented on YARN-2495: -- Hi [~Naganarasimha], What I meant is completely remove conf-based node label provider implementation (not leave a not-implemented class it in the patch), it should be fine to use a dummy node label provider in tests. Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script suggested by [~aw] (YARN-2729) ) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233644#comment-14233644 ] Wangda Tan commented on YARN-2495: -- [~Naganarasimha], Since size of the patch grows, and it will be hard for new people to review. I suggest to move conf-based node label provider implementation to a separated ticket under YARN-2492? And update title of this ticket accordingly. Thanks, Wangda Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml or using script suggested by [~aw]) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225432#comment-14225432
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683672/YARN-2495.20141126-1.patch
against trunk revision 78f7cdb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5936//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5936//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225439#comment-14225439 ] Wangda Tan commented on YARN-2495: -- [~Naganarasimha], bq. In this case i feel we need to start discussing on YARN-2728 : Support for disabling the Centralized NodeLabel validation in Distributed Node Label Configuration setup. Yes we need, that is originally why I created that ticket, thanks for taking that. Could you add your proposal to YARN-2728 first? We can go through it first before working on a patch. Thanks, Wangda Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml or using script suggested by [~aw]) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222424#comment-14222424 ] Wangda Tan commented on YARN-2495: -- Hi [~Naganarasimha], For 1#, sorry my bad, it has been more than half month since that comment :). I prefer use a consistent way, we have a isDecentralizedNodeLabelConfigurationEnabled, let's use it. Do you agree? Tests looks good to me. Thanks Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml or using script suggested by [~aw]) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222009#comment-14222009
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda], Thanks for the review
bq. 1) As same as ResourceTrackerService, it's better to have a field like
isDecentralizedNodeLabelConfigurationEnabled (or some other name you like) in
NodeStatusUpdaterImpl. Should be more clear than statement like
Earlier in one of your
[comment|https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14183135page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14183135],
you had mentioned as
{quote}
5) NodeStatusUpdaterImpl:
isDecentralizedNodeLabelsConf may not need here, if nodeLablesProvider passed
in is null. That means isDecentralizedNodeLabelsConf is false.
+nodeLabelsForHeartBeat = null;
+if (isDecentralizedNodeLabelsConf) {
...
{quote}
so had changed it , may be earlier you meant some thing else and i read the
comment wrongly ?
for 2 3 will rework and share the patch, but hope test cases are fine too ?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221342#comment-14221342
]
Wangda Tan commented on YARN-2495:
--
Hi [~Naganarasimha],
Thanks for update,
Several minor comments,
1) As same as ResourceTrackerService, it's better to have a field like
isDecentralizedNodeLabelConfigurationEnabled (or some other name you like) in
NodeStatusUpdaterImpl. Should be more clear than statement like
{code}
+if (nodeLabelsProvider!=null) {
{code}
2) In ResourceTrackerService,
The message:
{code}
String message =
NodeManager from node + host + (cmPort: + cmPort + httpPort:
+ httpPort + ) + registered with capability: + capability
-+ , assigned nodeId + nodeId;
++ , assigned nodeId + nodeId + , node labels {
++ StringUtils.join(,, nodeLabels) + } ;
{code}
Should add a check, only logging node labels message when replace is succeeded.
Ideally you should have a StringBuilder do this
3) A style suggestion is, as convention, bi-opts like =, !=, +, etc.
should leave a space between and after it. I can see several occurrences in the
patch.
Wangda
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219287#comment-14219287
]
Naganarasimha G R commented on YARN-2495:
-
Following 3 test failure issues does not seem to be introduced from of my
modifications
{quote}
TestApplicationClientProtocolOnHA.testGetContainersOnHA:154
TestApplicationClientProtocolOnHA.testSubmitApplicationOnHA:173
TestApplicationClientProtocolOnHA.testGetClusterMetricsOnHA:85
{quote}
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218439#comment-14218439
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12682448/YARN-2495.20141119-1.patch
against trunk revision 5bd048e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5880//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5880//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495.20141119-1.patch, YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206626#comment-14206626
]
Naganarasimha G R commented on YARN-2495:
-
{quote}
The benefit are
1) You don't have to update test cases for that
2) The semanic are clear, create a register request with label or not.
{quote}
True, and will be able to revert some unwanted testcase modification. have
corrected it.
bq. I suggest to have different option for script-based/config-based, even if
we can combine them together.
Ok, will have different config param for script and config based
bq. IIUC, NM_NODE_LABELS_FROM_CONFIG is a list of labels, even if we want to
separate the two properties, we cannot remove NM_NODE_LABELS_FROM_CONFIG,
correct?
Had searched it wrongly and as you mentioned the name of was not good enough
for me to recollect back too. corrected it
bq. I think it's better to leverage existing utility class instead of implement
your own. For example, you have set values but not check them, which is
incorrect, but using utility class can avoid such problem. Even if you added
new fields, tests will cover them without any changes:
Problem is ??TestPBImplRecords?? is in ??hadoop-yarn-common?? project and
??NodeHeartbeatRequestPBImpl?? and others are in ??hadoop-yarn-server-common??
project. So as we cant add dependency on ??hadoop-yarn-server-common?? in
??hadoop-yarn-common??, hence shall i create a new class extending
TestPBImplRecords in ??hadoop-yarn-server-common?? project. ?
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204974#comment-14204974
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda], Thanks for reviewing and sorry for the delayed reply ...
{quote}
Major comments:
1.
Makes sense to me, I suggest to add a field like node-labels-updated in
NodeHeartbeatRequest
{quote}
I have already added the below code in the NodeHeartbeatRequest interface for
this(or i dint get your comment correctly, please elaborate):
{quote}
@@ -26,7 +28,8 @@
public static NodeHeartbeatRequest newInstance(NodeStatus nodeStatus,
MasterKey lastKnownContainerTokenMasterKey,
- MasterKey lastKnownNMTokenMasterKey) {
+ MasterKey lastKnownNMTokenMasterKey, SetString nodeLabels,
+ boolean isNodeLabelsUpdated
@@ -45,4 +50,10 @@ public static NodeHeartbeatRequest newInstance(NodeStatus
nodeStatus,
+ public abstract boolean isNodeLabelsUpdated();
+ public abstract void setNodeLabelsUpdated(boolean isNodeLabelsUpdated);
{quote}
{quote}
3.Suggest to create a overload newInstance function without labels for
RegisterNodeManagerRequest to avoid check like:
(nodeLabelsProvider == null) ? null : nodeLabels);
{quote}
This if check will be there in the overloaded case to identify which overloaded
method to choose right ? i am not able to see any benefit from this.
{quote}
4.YarnConfiguration:
NM_NODE_LABELS_FROM_CONFIG sounds like a boolean value to me, how about call it
NM_NODE_LABELS_PREFIX + config-based + .node-labels?
NM_NODE_LABELS_FETCH_INTERVAL_MS should also be a part of config-based.
{quote}
NM_NODE_LABELS_FETCH_INTERVAL_MS DEFAULT_NM_NODE_LABELS_FETCH_INTERVAL_MS is
used in both the script based and also config based hence had not moved it
under config-based ?
whats your opinion having two properties separately for script and config based
is better? if reuse then can delete NM_NODE_LABELS_FROM_CONFIG, as its not used.
{quote}
5. PB tests
I think you can leverage TestPBImplRecords do all PB related tests, does it
enough?
{quote}
These i wanted to discuss with you , based on your patch changes for labels had
figured out this class but as i was modifying the existing PB class awas
wondering why these existing PB's are not added here.
Others comments have reworked, after clarifications of the above points will
upload the patch tomorrow...
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205689#comment-14205689
]
Wangda Tan commented on YARN-2495:
--
bq. I have already added the below code in the NodeHeartbeatRequest interface
for this(or i dint get your comment correctly, please elaborate):
I think I've misreading your patch, your patch should have done what we
discusse.
bq. This if check will be there in the overloaded case to identify which
overloaded method to choose right ? i am not able to see any benefit from this.
The benefit are
1) You don't have to update test cases for that
2) The semanic are clear, create a register request with label or not.
bq. NM_NODE_LABELS_FETCH_INTERVAL_MS DEFAULT_NM_NODE_LABELS_FETCH_INTERVAL_MS
is used in both the script based and also config based hence had not moved it
under config-based ?
I suggest to have different option for script-based/config-based, even if we
can combine them together.
bq. if reuse then can delete NM_NODE_LABELS_FROM_CONFIG, as its not used.
IIUC, NM_NODE_LABELS_FROM_CONFIG is a list of labels, even if we want to
separate the two properties, we cannot remove NM_NODE_LABELS_FROM_CONFIG,
correct?
(But I still suggest you to change it to: NM_NODE_LABELS_PREFIX +
config-based + .node-labels.)
bq. These i wanted to discuss with you , based on your patch changes for labels
had figured out this class but as i was modifying the existing PB class awas
wondering why these existing PB's are not added here.
I think it's better to leverage existing utility class instead of implement
your own. For example, you have set values but not check them, which is
incorrect, but using utility class can avoid such problem. Even if you added
new fields, tests will cover them without any changes:
{code}
-ApplicationId.newInstance(1234L, 2)));
+ApplicationId.newInstance(1234L, 2)),new HashSetString());
{code}
Will include review of unit tests after you uploaded new patch.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199494#comment-14199494
]
Wangda Tan commented on YARN-2495:
--
*Major comments:*
1.
As I commented,
https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14184146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14184146,
first line.
I think the statement
bq. if number of labels are more than the weight of heartbeat message will be
more and if the cluster nodes are more than more network IO, checking of labels
from Prev state to current state for all nodes is done in RM in earlier method
each NM was taking care of it self. Lot of read locks needs to be held @
NodesLabelsManager
Makes sense to me, I suggest to add a field like node-labels-updated in
NodeHeartbeatRequest
*Minor comments:*
1.
bq. static NodeLabelsProviderService createNodeLabelsProviderService
It's better to make this not static or protected in case the tests can override
it. Since it is only used by test, I don't suggest to make it static for that.
2.
NodeStatusUpdater:
2.1
{code}
+@SuppressWarnings(unchecked)
+SetString nodeLabels =
+(nodeLabelsProvider == null) ? Collections.EMPTY_SET
{code}
You can use CommonNodeLabelsManager##EMPTY_STRING_SET to avoid the warning
2.2
It's better to name areLabelsSameAsPrevOutPut - areNodeLabelsUnchanged, it's
shorter and has same semantics of nodeLabelsUpdated.
2.3
Need handle null value for {{areNodeLabelsUnchanged}} (or
{{areLabelsSameAsPrevOutPut}} previously). Since others can implement a new
NodeLabelsProvider
2.4
bq. +SetString nodeLabelsForHeartBeat = null;
Shoud put in while (...) { }
3.
Suggest to create a overload newInstance function without labels for
RegisterNodeManagerRequest to avoid check like:
{code}
+(nodeLabelsProvider == null) ? null : nodeLabels);
{code}
4.
YarnConfiguration:
- ENABLE_DECENTRALIZED_NODELABEL_CONFIGURATION -
DECENTRALIZED_NODE_LABELS_CONFIGURATION_ENABLED
- labels-provider.class - node-labels-provider.class
- For configuration file based options, I suggest to add a same prefix for
them, such as:
NM_NODE_LABELS_PREFIX + config-based
- NM_NODE_LABELS_FROM_CONFIG sounds like a boolean value to me, how about call
it NM_NODE_LABELS_PREFIX + config-based + .node-labels?
- NM_NODE_LABELS_FETCH_INTERVAL_MS should also be a part of config-based.
- Remove + /** End of Configurations in NodeManager for NodeLabelsFeature*/
5. PB tests
I think you can leverage {{TestPBImplRecords}} do all PB related tests, does it
enough?
6. ConfigurationNodeLabelsProvider
- checkForNodeLabelModification: how about call it updateNodeLabelsFromConfig?
Since the checkForNodeLabelModification is more like a read-only operation to
me.
- Remove
{code}
+} else {
+ if (LOG.isDebugEnabled())
+LOG.debug(errorMsg.toString());
+}
{code}
Since the error message is handled by ConfigurationMonitorTimerTask
- {{+e.printStackTrace();}} to LOG.error(e)
7. NodeLabelsProviderService:
bq. @return
Need to be added for getNodeLabels, or javadocs warning will raise
8. Change the NodeHeartbeatResponse#getAcceptNodeLabels -
getIsNodeLabelsAcceptedByRM, same as RegisterResponse (Or some beter name you
can think about, since the getAcceptedNodeLabels is more like a list of labels)
9. ResourceTrackerService:
9.1
{code}
+LOG.info(Node Labels { + StringUtils.join(,, nodeLabels)
++ } of NodeManager from + host
++ is not properly configured: + ex.getMessage() + );
{code}
Again, should use error here
9.2
The node label update logic for register/hearbeat are very similar, could you
merge common part of them?
Will include review of tests in next turn.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191656#comment-14191656
]
Hadoop QA commented on YARN-2495:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678260/YARN-2495.20141031-1.patch
against trunk revision d1828d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.client.TestRMFailover
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5652//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5652//console
This message is automatically generated.
Allow admin specify labels from each NM (Distributed configuration)
---
Key: YARN-2495
URL: https://issues.apache.org/jira/browse/YARN-2495
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
YARN-2495_20141022.1.patch
Target of this JIRA is to allow admin specify labels in each NM, this covers
- User can set labels in each NM (by setting yarn-site.xml or using script
suggested by [~aw])
- NM will send labels to RM via ResourceTracker API
- RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184163#comment-14184163 ] Sunil G commented on YARN-2495: --- Hi [~Naganarasimha Garla] [~gp.leftnoteasy] I have one comment here. bq. if exception raise when replace labels on node, put the new labels I assume current idea is in such a way that if one label from NM is found as invalid, complete labels sent by NM in that heartbeat will be considered as error. Could it be done in such a way that, give a number of labels from NM as 5 at a given heartbeat. In that 2 labels are found to be invalid by RM. If so, why could we accept the 3 passed labels, and send back 2 rejected labels back to NM. This may make the work better and easier to handle. How do you feel? Allow admin specify labels from each NM (Distributed configuration) --- Key: YARN-2495 URL: https://issues.apache.org/jira/browse/YARN-2495 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495_20141022.1.patch Target of this JIRA is to allow admin specify labels in each NM, this covers - User can set labels in each NM (by setting yarn-site.xml or using script suggested by [~aw]) - NM will send labels to RM via ResourceTracker API - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
