[jira] [Updated] (YARN-1457) YARN single node install issues on mvn clean install assembly:assembly on mapreduce project
[ https://issues.apache.org/jira/browse/YARN-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated YARN-1457: -- Component/s: (was: documentation) site YARN single node install issues on mvn clean install assembly:assembly on mapreduce project --- Key: YARN-1457 URL: https://issues.apache.org/jira/browse/YARN-1457 Project: Hadoop YARN Issue Type: Bug Components: site Affects Versions: 2.0.5-alpha Reporter: Rekha Joshi Priority: Minor Labels: mvn Attachments: yarn-mvn-mapreduce.txt YARN single node install - http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html On Mac OSX 10.7.3, Java 1.6, Protobuf 2.5.0 and hadoop-2.0.5-alpha.tar, mvn clean install -DskipTests succeds after a YARN fix on pom.xml(using 2.5.0 protobuf) But on hadoop-mapreduce-project mvn install fails for tests with below errors $ mvn clean install assembly:assembly -Pnative errors as in atatched yarn-mvn-mapreduce,txt On $mvn clean install assembly:assembly -DskipTests Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.410s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [13.781s] [INFO] hadoop-mapreduce-client-common SUCCESS [8.486s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [0.774s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [4.409s] [INFO] hadoop-mapreduce-client-hs SUCCESS [1.618s] [INFO] hadoop-mapreduce-client-jobclient . SUCCESS [4.470s] [INFO] hadoop-mapreduce-client-hs-plugins SUCCESS [0.561s] [INFO] Apache Hadoop MapReduce Examples .. SUCCESS [1.620s] [INFO] hadoop-mapreduce .. FAILURE [10.107s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 49.606s [INFO] Finished at: Thu Nov 28 16:20:52 GMT+05:30 2013 [INFO] Final Memory: 34M/118M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.3:assembly (default-cli) on project hadoop-mapreduce: Error reading assemblies: No assembly descriptors found. - [Help 1] $mvn package -Pdist -DskipTests=true -Dtar works The documentation needs to be updated for possible issues and resolutions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1104) NMs to support rolling logs of stdout stderr
[ https://issues.apache.org/jira/browse/YARN-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871834#comment-13871834 ] Steve Loughran commented on YARN-1104: -- in Hoya we're setting up log4j for rolling logs, both in the AM and in deployed HBase containers. We'll have to see how well this works with long-lived, busy clusters. YARN- highlights how the web UI can't handle big logs NMs to support rolling logs of stdout stderr -- Key: YARN-1104 URL: https://issues.apache.org/jira/browse/YARN-1104 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.1.0-beta Reporter: Steve Loughran Currently NMs stream the stdout and stderr streams of a container to a file. For longer lived processes those files need to be rotated so that the log doesn't overflow -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871837#comment-13871837 ] Steve Loughran commented on YARN-1601: -- Makes sense. Let me do a branch-2 build before and after this patch see what comes out 3rd party JARs are missing from hadoop-dist output -- Key: YARN-1601 URL: https://issues.apache.org/jira/browse/YARN-1601 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-1601.patch With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/. We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn. As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly). Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871845#comment-13871845 ] Steve Loughran commented on YARN-1603: -- +1 Remove two *.orig files which were unexpectedly committed - Key: YARN-1603 URL: https://issues.apache.org/jira/browse/YARN-1603 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Minor Attachments: YARN-1603.1.patch FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
Junping Du created YARN-1604: Summary: Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig Key: YARN-1604 URL: https://issues.apache.org/jira/browse/YARN-1604 Project: Hadoop YARN Issue Type: Improvement Reporter: Junping Du Priority: Minor In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and FairScheduler.java.orig are get committed which is not necessary. Remove both of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
[ https://issues.apache.org/jira/browse/YARN-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-1604: - Attachment: YARN-1604.patch A simple fix to remove both files. Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig Key: YARN-1604 URL: https://issues.apache.org/jira/browse/YARN-1604 Project: Hadoop YARN Issue Type: Improvement Reporter: Junping Du Priority: Minor Attachments: YARN-1604.patch In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and FairScheduler.java.orig are get committed which is not necessary. Remove both of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
[ https://issues.apache.org/jira/browse/YARN-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-1604. -- Resolution: Duplicate Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig Key: YARN-1604 URL: https://issues.apache.org/jira/browse/YARN-1604 Project: Hadoop YARN Issue Type: Improvement Reporter: Junping Du Priority: Minor Attachments: YARN-1604.patch In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and FairScheduler.java.orig are get committed which is not necessary. Remove both of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871859#comment-13871859 ] Junping Du commented on YARN-1603: -- +1. I will commit it shortly. Remove two *.orig files which were unexpectedly committed - Key: YARN-1603 URL: https://issues.apache.org/jira/browse/YARN-1603 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Minor Attachments: YARN-1603.1.patch FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871906#comment-13871906 ] Hudson commented on YARN-1603: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4999 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4999/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Remove two *.orig files which were unexpectedly committed - Key: YARN-1603 URL: https://issues.apache.org/jira/browse/YARN-1603 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Minor Attachments: YARN-1603.1.patch FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[
https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871920#comment-13871920
]
Steve Loughran commented on YARN-1601:
--
before the patch
{code}
$ find hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-api-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-client-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-tests-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-api-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-client-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-client-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-common-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-common-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-common-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-common-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-tests-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//test
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//test/hadoop-yarn-server-tests-2.4.0-SNAPSHOT-tests.jar
{code}
3rd party JARs are missing from hadoop-dist output
--
Key: YARN-1601
URL: https://issues.apache.org/jira/browse/YARN-1601
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0, 2.4.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: YARN-1601.patch
With the build changes of YARN-888 we are leaving out all 3rd party JArs used
directly by YARN under /share/hadoop/yarn/lib/.
We did not notice this when running minicluster because they all happen to be
in the classpath from hadoop-common and hadoop-yarn.
As 3d party JARs are not 'public' interfaces we cannot rely on them being
provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd
party dependency that yarn uses this would break yarn if yarn does not pull
that dependency explicitly).
Also, this will
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[
https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871921#comment-13871921
]
Steve Loughran commented on YARN-1601:
--
after the patch
{code}
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//.DS_Store
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-api-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-client-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-tests-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/activation-1.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/aopalliance-1.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/asm-3.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-cli-1.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-codec-1.4.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-compress-1.4.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-httpclient-3.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-io-2.4.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-lang-2.6.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-logging-1.1.3.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guava-11.0.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guice-3.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guice-servlet-3.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-core-asl-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-jaxrs-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-mapper-asl-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-xc-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/javax.inject-1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jaxb-api-2.2.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jaxb-impl-2.2.3-1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-client-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-core-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-guice-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-json-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-server-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jettison-1.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jetty-6.1.26.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jetty-util-6.1.26.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jline-0.9.94.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jsr305-1.3.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/log4j-1.2.17.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/protobuf-java-2.5.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/servlet-api-2.5.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/stax-api-1.0-2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/stax-api-1.0.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/xz-1.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/zookeeper-3.4.5.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-api-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-sources.jar
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871923#comment-13871923 ] Steve Loughran commented on YARN-1601: -- confirmed that problem exists patch does as claimed, +1 3rd party JARs are missing from hadoop-dist output -- Key: YARN-1601 URL: https://issues.apache.org/jira/browse/YARN-1601 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-1601.patch With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/. We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn. As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly). Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871940#comment-13871940 ] Hudson commented on YARN-1603: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5000 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5000/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558340) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java.orig * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java.orig svn merge -c -1558337 . for reverting YARN-1603, which only partly committed. (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558339) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Remove two *.orig files which were unexpectedly committed - Key: YARN-1603 URL: https://issues.apache.org/jira/browse/YARN-1603 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Minor Fix For: 2.4.0 Attachments: YARN-1603.1.patch FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872051#comment-13872051 ] Hudson commented on YARN-1598: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1670/]) YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558251) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java HA-related rmadmin commands don't work on a secure cluster -- Key: YARN-1598 URL: https://issues.apache.org/jira/browse/YARN-1598 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Fix For: 2.4.0 Attachments: yarn-1598-1.patch The HA-related commands like -getServiceState -checkHealth etc. don't work in a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872044#comment-13872044 ] Hudson commented on YARN-1567: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1645/]) YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558228) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload - Key: YARN-1567 URL: https://issues.apache.org/jira/browse/YARN-1567 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, YARN-1567.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872235#comment-13872235 ] Sean Mackrory commented on YARN-1601: - +1 (non-committer) - I can confirm the problem from a Bigtop perspective. If you were to try and build the hadoop-client package it would break because there wouldn't be any other Hadoop package that would provide one of the dependencies Maven determined was needed for a YARN client (specifically, stax-api 1.0-2, which has a different group ID than stax-api 1.0.1). 3rd party JARs are missing from hadoop-dist output -- Key: YARN-1601 URL: https://issues.apache.org/jira/browse/YARN-1601 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-1601.patch With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/. We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn. As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly). Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872270#comment-13872270 ] Alejandro Abdelnur commented on YARN-1601: -- Thanks Steve. Thanks Sean, regarding the stax-api JAR, I've opened another JIRA to fix that, HADOOP-10235. 3rd party JARs are missing from hadoop-dist output -- Key: YARN-1601 URL: https://issues.apache.org/jira/browse/YARN-1601 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-1601.patch With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/. We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn. As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly). Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872287#comment-13872287 ] Hudson commented on YARN-1601: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5001 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5001/]) YARN-1601. 3rd party JARs are missing from hadoop-dist output. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558465) * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/pom.xml 3rd party JARs are missing from hadoop-dist output -- Key: YARN-1601 URL: https://issues.apache.org/jira/browse/YARN-1601 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-1601.patch With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/. We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn. As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly). Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
[ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872300#comment-13872300 ] Robert Joseph Evans commented on YARN-1530: --- I agree that we need to think about load and plan for something that can handle at least 20x the current load but preferably 100x. However, I am not that sure that the load will be a huge problem at least for current MR clusters. We have seen very large jobs as well, but 700 MB history file job does not finish instantly. I took a look at a 3500 node cluster we have that is under fairly heavy load, and looking at the done directory for yesterday, I saw what amounted to about 1.7MB/sec of data on average. Gigabit Ethernet should be able to handle 15 to 20 times this (assuming that we read as much data as we write, and that the storage may require some replication). I am fine with the proposed solution by [~lohit] so long as the history service always provides a restful interface and the AM can decide if it wants to use it, or go through a different higher load channel. Otherwise non-java based AMs would not necessarily be able to write to the history service. I am also a bit nervous about using the history service for recovery or as a backend for the current MR APIs if we have a pub/sub system as a link between the applications and the history service. I don't think it is a show stopper, it just opens the door for a number of corner cases that will have to be dealt with, like an MR AM crashes badly and the client goes to the history service to get the counters/etc, when does the history service know that all of the events for the MR AM have been processed so it can return those counters, or perhaps other data? I am not totally sure what data may be a show stopper for this, but the lag means all applications have to be sure that they don't use the history service for split brain problems or things like that. [Umbrella] Store, manage and serve per-framework application-timeline data -- Key: YARN-1530 URL: https://issues.apache.org/jira/browse/YARN-1530 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Attachments: application timeline design-20140108.pdf This is a sibling JIRA for YARN-321. Today, each application/framework has to do store, and serve per-framework data all by itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management and serving of per-framework data from various applications, both running and finished. The aim is to change YARN to collect and store data in a generic manner with plugin points for frameworks to do their own thing w.r.t interpretation and serving. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1584) Support explicit failover when automatic failover is enabled
[ https://issues.apache.org/jira/browse/YARN-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872514#comment-13872514 ] Bikas Saha commented on YARN-1584: -- And what use case does this solve other than make RM1 become standby? Downside is a lot of churn in client that are currently using YARN as they switch from RM1 to RM2? If RM1 is going to go back and join the election anyways then why are we doing this? Secondly, I dont see how this works when there are 3 RMs in the leader election. I dont think we want to restrict our model to only support 2 RMs. Lastly, isnt this something that can be done entirely on the client side. Client would first call transitionToStandby() on all RM's other than the one that the admin want to make active. It will determine if the desired RM has become active (or timeout) and then make the remaining RMs' transitionToActive() and rejoin the election. Support explicit failover when automatic failover is enabled Key: YARN-1584 URL: https://issues.apache.org/jira/browse/YARN-1584 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla YARN-1029 adds automatic failover support. However, users can't explicitly ask for a failover from one RM to the other without stopping the other RM. Stopping the RM until the other RM takes over and then restarting the first RM is more involving and exposes the RM-ensemble to SPOF for a longer duration. It would be nice to allow explicit failover through yarn rmadmin -failover command. PS: HDFS supports -failover option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1595) Test failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1595: -- Attachment: YARN-1595-20140115.txt Here's a patch that addresses all test failures. The core change is to make application-history-service optional at both the RM as well as the client. Test failures on YARN-321 branch Key: YARN-1595 URL: https://issues.apache.org/jira/browse/YARN-1595 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1595-20140115.txt mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1587) [YARN-321] MERGE Patch for YARN-321
[ https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1587: -- Attachment: YARN-1587-20140115.txt Updated patch = all YARN-321 changes + YARN-1595. [YARN-321] MERGE Patch for YARN-321 --- Key: YARN-1587 URL: https://issues.apache.org/jira/browse/YARN-1587 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Vinod Kumar Vavilapalli Fix For: YARN-321 Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt, YARN-1587-20140115.txt, YARN-321-merge-1.patch Merge Patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[ https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1587: -- Summary: [YARN-321] Merge Patch for YARN-321 (was: [YARN-321] MERGE Patch for YARN-321) [YARN-321] Merge Patch for YARN-321 --- Key: YARN-1587 URL: https://issues.apache.org/jira/browse/YARN-1587 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Vinod Kumar Vavilapalli Fix For: YARN-321 Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt, YARN-1587-20140115.txt, YARN-321-merge-1.patch Merge Patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1595) Test failures on YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872569#comment-13872569
]
Hadoop QA commented on YARN-1595:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623213/YARN-1595-20140115.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2889//console
This message is automatically generated.
Test failures on YARN-321 branch
Key: YARN-1595
URL: https://issues.apache.org/jira/browse/YARN-1595
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-1595-20140115.txt
mvn test doesn't pass on YARN-321 branch anymore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1605) Fix formatting issues with new module in YARN-321 branch
Vinod Kumar Vavilapalli created YARN-1605: - Summary: Fix formatting issues with new module in YARN-321 branch Key: YARN-1605 URL: https://issues.apache.org/jira/browse/YARN-1605 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli There are a bunch of formatting issues. I'm restricting myself for a sweep of all the files in the new module. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1600) RM does not startup when security is enabled without spnego configured
[ https://issues.apache.org/jira/browse/YARN-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872586#comment-13872586 ] Haohui Mai commented on YARN-1600: -- I think at least in the short term that the earlier approach of YARN-1463 could work. We'll need to revisit the issue of supporting SPNEGO and third-party HTTP authentication in the longer term. RM does not startup when security is enabled without spnego configured -- Key: YARN-1600 URL: https://issues.apache.org/jira/browse/YARN-1600 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Priority: Blocker We have a custom auth filter in front of our various UI pages that handles user authentication. However currently the RM assumes that if security is enabled then the user must have configured spnego as well for the RM web pages which is not true in our case. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1595) Test failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1595: -- Attachment: YARN-1595-20140115.1.txt Tx for the review Zhijie! Here's an updated patch with the yarn-default.xml changes. Test failures on YARN-321 branch Key: YARN-1595 URL: https://issues.apache.org/jira/browse/YARN-1595 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1595-20140115.1.txt, YARN-1595-20140115.txt mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[
https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Weitz updated YARN-1351:
---
Description:
While trying to print a warning, two values of the wrong type (Resource instead
of int) are passed into a String.format method call, leading to a runtime
exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into
each other, either because the number of virtual cores or the memory is too
small. I changed the %d's into %s, this way the warning will contain both the
cores and the memory.
was:
While trying to print a warning, two values of the wrong type (Resource instead
of int) are passed into a String.format method call, leading to a runtime
exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into
each other, either because the number of virtual cores or the memory is too
small. I changed the %d's into %s, this way the warning will contain both the
cores and the memory.
Following the patch that fixes the issue:
Index:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
===
---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
(revision 1535589)
+++
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
(working copy)
@@ -471,7 +471,7 @@
if (maxQueueResources.containsKey(queueName)
minQueueResources.containsKey(queueName)
!Resources.fitsIn(minQueueResources.get(queueName),
maxQueueResources.get(queueName))) {
- LOG.warn(String.format(Queue %s has max resources %d less than min
resources %d,
+ LOG.warn(String.format(Queue %s has max resources %s less than min
resources %s,
queueName, maxQueueResources.get(queueName),
minQueueResources.get(queueName)));
}
}
Invalid string format in Fair Scheduler log warn message
Key: YARN-1351
URL: https://issues.apache.org/jira/browse/YARN-1351
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Konstantin Weitz
Attachments: fixprnt.patch
While trying to print a warning, two values of the wrong type (Resource
instead of int) are passed into a String.format method call, leading to a
runtime exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into
each other, either because the number of virtual cores or the memory is too
small. I changed the %d's into %s, this way the warning will contain both the
cores and the memory.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872698#comment-13872698 ] Konstantin Weitz commented on YARN-1351: Because of the simplicity of the fix, writing a test case would not be worthwhile. Because of the locality and size of the change, I used manual inspection to verify that the patch fixes the problem and does not introduce new problems. Sandy Ryza, do agree with this reasoning, or do you have any suggestions on how to improve the patch? Invalid string format in Fair Scheduler log warn message Key: YARN-1351 URL: https://issues.apache.org/jira/browse/YARN-1351 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Konstantin Weitz Attachments: fixprnt.patch While trying to print a warning, two values of the wrong type (Resource instead of int) are passed into a String.format method call, leading to a runtime exception, in the file: _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. The warning was intended to be printed whenever the resources don't fit into each other, either because the number of virtual cores or the memory is too small. I changed the %d's into %s, this way the warning will contain both the cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872709#comment-13872709 ] Karthik Kambatla commented on YARN-1410: bq. But, I am not sure why we need to use these annotation in RM restart. Adding to what Bikas has already said. Consider RM restarting while an app submission is in-flight. The possibilities are exactly the same as in the case of failover. bq. Also, the AtMostOnce and Idempotent annotation are only used when RetryDecision is FAILOVER_AND_RETRY. So, this is another reason why we do not have them in RM restart case (For the RM restart, the valid RetryDecision is RETRY). Precisely my point. IMO, RM restart should not be directly using RetryUptoFixedTime or whatever it is currently using. Eventually, when RetryPolicy#shouldRetry is called, the annotations should be taken into account. Handle client failover during 2 step client API's like app submission - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[
https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872707#comment-13872707
]
Hadoop QA commented on YARN-1587:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623214/YARN-1587-20140115.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryClientService
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2887//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2887//console
This message is automatically generated.
[YARN-321] Merge Patch for YARN-321
---
Key: YARN-1587
URL: https://issues.apache.org/jira/browse/YARN-1587
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Vinod Kumar Vavilapalli
Fix For: YARN-321
Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt,
YARN-1587-20140115.txt, YARN-321-merge-1.patch
Merge Patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[
https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872705#comment-13872705
]
Hadoop QA commented on YARN-1587:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623214/YARN-1587-20140115.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryClientService
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
The following test timeouts occurred in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2888//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2888//console
This message is automatically generated.
[YARN-321] Merge Patch for YARN-321
---
Key: YARN-1587
URL: https://issues.apache.org/jira/browse/YARN-1587
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Vinod Kumar Vavilapalli
Fix For: YARN-321
Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt,
YARN-1587-20140115.txt, YARN-321-merge-1.patch
Merge Patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1595) Test failures on YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872722#comment-13872722
]
Hadoop QA commented on YARN-1595:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623245/YARN-1595-20140115.1.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2890//console
This message is automatically generated.
Test failures on YARN-321 branch
Key: YARN-1595
URL: https://issues.apache.org/jira/browse/YARN-1595
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-1595-20140115.1.txt, YARN-1595-20140115.txt
mvn test doesn't pass on YARN-321 branch anymore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1606) yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java
[ https://issues.apache.org/jira/browse/YARN-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872725#comment-13872725 ] Sandy Ryza commented on YARN-1606: -- +1 yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java --- Key: YARN-1606 URL: https://issues.apache.org/jira/browse/YARN-1606 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Labels: conf, newbie Attachments: yarn-1606-1.patch Looks like YARN-1029 changed at one location, but not the other. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872767#comment-13872767 ] Sandy Ryza commented on YARN-1351: -- (Sorry for the delay) Invalid string format in Fair Scheduler log warn message Key: YARN-1351 URL: https://issues.apache.org/jira/browse/YARN-1351 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Konstantin Weitz Assignee: Konstantin Weitz Fix For: 2.4.0 Attachments: fixprnt.patch While trying to print a warning, two values of the wrong type (Resource instead of int) are passed into a String.format method call, leading to a runtime exception, in the file: _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. The warning was intended to be printed whenever the resources don't fit into each other, either because the number of virtual cores or the memory is too small. I changed the %d's into %s, this way the warning will contain both the cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1351: - Assignee: Konstantin Weitz Invalid string format in Fair Scheduler log warn message Key: YARN-1351 URL: https://issues.apache.org/jira/browse/YARN-1351 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Konstantin Weitz Assignee: Konstantin Weitz Fix For: 2.4.0 Attachments: fixprnt.patch While trying to print a warning, two values of the wrong type (Resource instead of int) are passed into a String.format method call, leading to a runtime exception, in the file: _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. The warning was intended to be printed whenever the resources don't fit into each other, either because the number of virtual cores or the memory is too small. I changed the %d's into %s, this way the warning will contain both the cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1573) ZK store should pick a private password for root-node-acls
[
https://issues.apache.org/jira/browse/YARN-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872769#comment-13872769
]
Hadoop QA commented on YARN-1573:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623242/yarn-1573-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2891//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2891//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2891//console
This message is automatically generated.
ZK store should pick a private password for root-node-acls
--
Key: YARN-1573
URL: https://issues.apache.org/jira/browse/YARN-1573
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: yarn-1573-1.patch
Currently, when HA is enabled, ZK store uses cluster-timestamp as the
password for root node ACLs to give the Active RM exclusive access to the
store. A more private value like a random number might be better.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872786#comment-13872786 ] Hudson commented on YARN-1351: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5006 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5006/]) YARN-1351. Invalid string format in Fair Scheduler log warn message (Konstantin Weitz via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1558597) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java Invalid string format in Fair Scheduler log warn message Key: YARN-1351 URL: https://issues.apache.org/jira/browse/YARN-1351 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Konstantin Weitz Assignee: Konstantin Weitz Fix For: 2.4.0 Attachments: fixprnt.patch While trying to print a warning, two values of the wrong type (Resource instead of int) are passed into a String.format method call, leading to a runtime exception, in the file: _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. The warning was intended to be printed whenever the resources don't fit into each other, either because the number of virtual cores or the memory is too small. I changed the %d's into %s, this way the warning will contain both the cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872937#comment-13872937 ] Alejandro Abdelnur commented on YARN-1608: -- +1 LinuxContainerExecutor has a few DEBUG messages at INFO level - Key: YARN-1608 URL: https://issues.apache.org/jira/browse/YARN-1608 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial Labels: log Attachments: yarn-1608-1.patch LCE has a few INFO level log messages meant to be at debug level. In fact, they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872938#comment-13872938 ] Alejandro Abdelnur commented on YARN-1608: -- +1 LinuxContainerExecutor has a few DEBUG messages at INFO level - Key: YARN-1608 URL: https://issues.apache.org/jira/browse/YARN-1608 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial Labels: log Attachments: yarn-1608-1.patch LCE has a few INFO level log messages meant to be at debug level. In fact, they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: YARN-1474.2.patch Updated a patch based on Karthik's approach. Sorry for delaying. I needed to redesign reinitialize() API and design what we should do in serviceInit/serviceStop/serviceStart APIs to take this ticket. Make schedulers services Key: YARN-1474 URL: https://issues.apache.org/jira/browse/YARN-1474 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Affects Versions: 2.4.0 Reporter: Sandy Ryza Assignee: Tsuyoshi OZAWA Attachments: YARN-1474.1.patch, YARN-1474.2.patch Schedulers currently have a reinitialize but no start and stop. Fitting them into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1474) Make schedulers services
[
https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873117#comment-13873117
]
Hadoop QA commented on YARN-1474:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623327/YARN-1474.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2894//console
This message is automatically generated.
Make schedulers services
Key: YARN-1474
URL: https://issues.apache.org/jira/browse/YARN-1474
Project: Hadoop YARN
Issue Type: Sub-task
Components: scheduler
Affects Versions: 2.4.0
Reporter: Sandy Ryza
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1474.1.patch, YARN-1474.2.patch
Schedulers currently have a reinitialize but no start and stop. Fitting them
into the YARN service model would make things more coherent.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873123#comment-13873123 ] Tsuyoshi OZAWA commented on YARN-1474: -- Main updates are as follows: 1. Made AbstractYarnScheduler AbstractService. 2. reinitialize() API just executes reloading configuration or reset initialization, because this API is called from AdminService#refreshQueues(). 3. setRMContext() API is for initializing RMContext in schedulers. This is because we can only pass Configuration to serviceInit() API. 4. Scheduers' serviceInit(Configuration conf) execute initialization of configurations or threads. 5. Scheduler's serviceStart() start threads. 6. Scheduler's serviceStop() don't anything currently. Just calling super.serviceStop(). [~kkambatl] and [~vinodkv], do you have feedbacks? Make schedulers services Key: YARN-1474 URL: https://issues.apache.org/jira/browse/YARN-1474 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Affects Versions: 2.4.0 Reporter: Sandy Ryza Assignee: Tsuyoshi OZAWA Attachments: YARN-1474.1.patch, YARN-1474.2.patch Schedulers currently have a reinitialize but no start and stop. Fitting them into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
