Allen Wittenauer commented on YARN-2495:

I don't fully understand your question, but I'll admit I've been distracted 
with other JIRAs lately.  Hopefully I'm guessing correctly though... :)

The idea that there are 'valid' labels and that one turns them on/off seems to 
conflict with the goals that I want to see fulfilled with this feature.  Let me 
give a more concrete example, since I know a lot of people are confused as to 
why labels should be dynamic anyway:

Here's node A's configuration:

$ hadoop version
Hadoop 2.6.0
Subversion http://github.com/apache/hadoop -r 
Compiled by aw on 2014-10-10T19:31Z
Compiled with protoc 2.5.0
>From source with checksum 201dad5b10939faa6e5841378c8c94
This command was run using 
$ java -version
java version "1.6.0_32"
OpenJDK Runtime Environment (IcedTea6 1.13.4) (rhel-
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Here's node B's configuration:
$ ~/HADOOP/hadoop-3.0.0-SNAPSHOT/bin/hadoop version
Hadoop 3.0.0
Source code repository https://git-wip-us.apache.org/repos/asf/hadoop.git -r 
Compiled by aw on 2014-10-08T21:12Z
Compiled with protoc 2.5.0
>From source with checksum ae703b1a38a35d19f4584495dc31944
This command was run using 
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

(Ignore the fact that they probably can't be in the same cluster. That's not 
really relevant.)

I want to provide a script similar to this one:
#!/usr/bin/env bash

HADOOPVERSION=$(${HADOOP_PREFIX}/bin/hadoop version 2>/dev/null | /usr/bin/head 
-1 |/usr/bin/cut -f2 -d\ )
JAVAVERSION=$(${JAVA_HOME}/bin/java -version 2>&1 |/usr/bin/head -1| 
/usr/bin/cut -f2 -d\" )
JAVAMMM=$(echo ${JAVAVERSION}| /usr/bin/cut -f1 -d_)


This script, when run, should tell the RM that:

Node A has labels 1.6.0_32, jdk1.6.0, and hadoop_2.6.0 .
Node B has labels LABELS=jdk1.7.0_67, jdk1.7.0, and hadoop_3.0.0

Users should be able to submit a job that specifies that it only runs with 
'hadoop_3.0.0'.  Or that it requires 'jdk_1.7.0'.

Making labels either valid or invalid based upon a central configuration 
defeats the purpose of having a living, breathing cluster able to adapt to 
change without operations intervention.  If we are rolling out a new version of 
the JDK, I shouldn't have to tell the system that it's ok to broadcast that JDK 
version first.

> Allow admin specify labels in each NM (Distributed configuration)
> -----------------------------------------------------------------
>                 Key: YARN-2495
>                 URL: https://issues.apache.org/jira/browse/YARN-2495
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
> Target of this JIRA is to allow admin specify labels in each NM, this covers
> - User can set labels in each NM (by setting yarn-site.xml or using script 
> suggested by [~aw])
> - NM will send labels to RM via ResourceTracker API
> - RM will set labels in NodeLabelManager when NM register/update labels

This message was sent by Atlassian JIRA

Reply via email to