[jira] [Commented] (YARN-2669) FairScheduler: queueName shouldn't allow periods the allocation.xml
[
https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184419#comment-14184419
]
bc Wong commented on YARN-2669:
---
AllocationFileLoaderService.java
* When throwing an error, would also output the offending queueName.
QueuePlacementRule.java
* Would log when you convert the username to one that doesn't have a queue.
* I'm worried about username conflicts after the conversion, e.g. "eric.koffee"
== "erick.offee". Replacing the dot with something else helps, but doesn't
eliminate the problem. You'd need some escaping rule, like replacing any
naturally occurring single underscore with two underscores, and then replacing
a dot with a single underscore.
> FairScheduler: queueName shouldn't allow periods the allocation.xml
> ---
>
> Key: YARN-2669
> URL: https://issues.apache.org/jira/browse/YARN-2669
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: YARN-2669-1.patch, YARN-2669-2.patch
>
>
> For an allocation file like:
> {noformat}
>
>
> 4096mb,4vcores
>
>
> {noformat}
> Users may wish to config minResources for a queue with full path "root.q1".
> However, right now, fair scheduler will treat this configureation for the
> queue with full name "root.root.q1". We need to print out a warning msg to
> notify users about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184466#comment-14184466 ] Hudson commented on YARN-1492: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/724/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copying of jobjars and libjars becomes so > prevalent that it consumes a large portion of the network bandwidth, not to > speak of defeating the purpose of "bringing compute to where data is". This > is wasteful because in most cases code doesn't change much across many jobs. > I'd like to propose and discuss feasibility of introducing a truly shared > cache so that multiple jobs from multiple users can share and cache jars. > This JIRA is to open the discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2723) rmadmin -replaceLabelsOnNode does not correctly parse port
[
https://issues.apache.org/jira/browse/YARN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184467#comment-14184467
]
Hudson commented on YARN-2723:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/724/])
YARN-2723. Fix rmadmin -replaceLabelsOnNode does not correctly parse (xgong:
rev 65d95b1a520d4ffdf024dbdfcf11d855a3948056)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
> rmadmin -replaceLabelsOnNode does not correctly parse port
> --
>
> Key: YARN-2723
> URL: https://issues.apache.org/jira/browse/YARN-2723
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Phil D'Amore
>Assignee: Naganarasimha G R
> Fix For: 2.6.0
>
> Attachments: YARN-2723.20141023.1.patch, yarn-2723.20141023.2.patch
>
>
> There is an off-by-one issue in RMAdminCLI.java (line 457):
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")));
> should probably be:
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")+1));
> Currently attempting to add a label to a node with a port specified looks
> like this:
> [yarn@ip-10-0-0-66 ~]$ yarn rmadmin -replaceLabelsOnNode
> node.example.com:45454,test-label
> replaceLabelsOnNode: For input string: ":45454"
> Usage: yarn rmadmin [-replaceLabelsOnNode [node1:port,label1,label2
> node2:port,label1,label2]]
> It appears to be trying to parse the ':' as part of the integer because the
> substring index is off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2183) Cleaner service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184468#comment-14184468 ] Hudson commented on YARN-2183: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/724/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java > Cleaner service for cache manager > - > > Key: YARN-2183 > URL: https://issues.apache.org/jira/browse/YARN-2183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2183-trunk-v1.patch, YARN-2183-trunk-v2.patch, > YARN-2183-trunk-v3.patch, YARN-2183-trunk-v4.patch, YARN-2183-trunk-v5.patch, > YARN-2183-trunk-v6.patch, YARN-2183-trunk-v7.patch, YARN-2183-trunk-v8.patch, > on-demand-cleaner.patch > > > Implement the cleaner service for the cache manager along with metrics for > the service. This service is responsible for cleaning up old resource > references in the manager and removing stale entries from the cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184477#comment-14184477
]
Varun Vasudev commented on YARN-2734:
-
+1 - patch looks good.
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2183) Cleaner service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184499#comment-14184499 ] Hudson commented on YARN-2183: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java > Cleaner service for cache manager > - > > Key: YARN-2183 > URL: https://issues.apache.org/jira/browse/YARN-2183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2183-trunk-v1.patch, YARN-2183-trunk-v2.patch, > YARN-2183-trunk-v3.patch, YARN-2183-trunk-v4.patch, YARN-2183-trunk-v5.patch, > YARN-2183-trunk-v6.patch, YARN-2183-trunk-v7.patch, YARN-2183-trunk-v8.patch, > on-demand-cleaner.patch > > > Implement the cleaner service for the cache manager along with metrics for > the service. This service is responsible for cleaning up old resource > references in the manager and removing stale entries from the cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2723) rmadmin -replaceLabelsOnNode does not correctly parse port
[
https://issues.apache.org/jira/browse/YARN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184498#comment-14184498
]
Hudson commented on YARN-2723:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/])
YARN-2723. Fix rmadmin -replaceLabelsOnNode does not correctly parse (xgong:
rev 65d95b1a520d4ffdf024dbdfcf11d855a3948056)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
> rmadmin -replaceLabelsOnNode does not correctly parse port
> --
>
> Key: YARN-2723
> URL: https://issues.apache.org/jira/browse/YARN-2723
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Phil D'Amore
>Assignee: Naganarasimha G R
> Fix For: 2.6.0
>
> Attachments: YARN-2723.20141023.1.patch, yarn-2723.20141023.2.patch
>
>
> There is an off-by-one issue in RMAdminCLI.java (line 457):
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")));
> should probably be:
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")+1));
> Currently attempting to add a label to a node with a port specified looks
> like this:
> [yarn@ip-10-0-0-66 ~]$ yarn rmadmin -replaceLabelsOnNode
> node.example.com:45454,test-label
> replaceLabelsOnNode: For input string: ":45454"
> Usage: yarn rmadmin [-replaceLabelsOnNode [node1:port,label1,label2
> node2:port,label1,label2]]
> It appears to be trying to parse the ':' as part of the integer because the
> substring index is off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184497#comment-14184497 ] Hudson commented on YARN-1492: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copying of jobjars and libjars becomes so > prevalent that it consumes a large portion of the network bandwidth, not to > speak of defeating the purpose of "bringing compute to where data is". This > is wasteful because in most cases code doesn't change much across many jobs. > I'd like to propose and discuss feasibility of introducing a truly shared > cache so that multiple jobs from multiple users can share and cache jars. > This JIRA is to open the discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2745) YARN new pluggable scheduler which does multi-resource packing
[ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184506#comment-14184506 ] Karthik Kambatla commented on YARN-2745: That is pretty interesting work. Thanks for filing this JIRA, [~rgrandl] FairScheduler allows setting a different policy per queue. Given fairness is one of your criteria, did you consider adding a policy to the FairScheduler instead of a completely new scheduler? > YARN new pluggable scheduler which does multi-resource packing > -- > > Key: YARN-2745 > URL: https://issues.apache.org/jira/browse/YARN-2745 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Reporter: Robert Grandl > Attachments: sigcomm_14_tetris_talk.pptx, tetris_design_doc.docx, > tetris_paper.pdf > > > In this umbrella JIRA we propose a new pluggable scheduler, which accounts > for all resources used by a task (CPU, memory, disk, network) and it is able > to achieve three competing objectives: fairness, improve cluster utilization > and reduces average job completion time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2723) rmadmin -replaceLabelsOnNode does not correctly parse port
[
https://issues.apache.org/jira/browse/YARN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184511#comment-14184511
]
Hudson commented on YARN-2723:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/])
YARN-2723. Fix rmadmin -replaceLabelsOnNode does not correctly parse (xgong:
rev 65d95b1a520d4ffdf024dbdfcf11d855a3948056)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
> rmadmin -replaceLabelsOnNode does not correctly parse port
> --
>
> Key: YARN-2723
> URL: https://issues.apache.org/jira/browse/YARN-2723
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Phil D'Amore
>Assignee: Naganarasimha G R
> Fix For: 2.6.0
>
> Attachments: YARN-2723.20141023.1.patch, yarn-2723.20141023.2.patch
>
>
> There is an off-by-one issue in RMAdminCLI.java (line 457):
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")));
> should probably be:
> port = Integer.valueOf(nodeIdStr.substring(nodeIdStr.indexOf(":")+1));
> Currently attempting to add a label to a node with a port specified looks
> like this:
> [yarn@ip-10-0-0-66 ~]$ yarn rmadmin -replaceLabelsOnNode
> node.example.com:45454,test-label
> replaceLabelsOnNode: For input string: ":45454"
> Usage: yarn rmadmin [-replaceLabelsOnNode [node1:port,label1,label2
> node2:port,label1,label2]]
> It appears to be trying to parse the ':' as part of the integer because the
> substring index is off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184510#comment-14184510 ] Hudson commented on YARN-1492: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copying of jobjars and libjars becomes so > prevalent that it consumes a large portion of the network bandwidth, not to > speak of defeating the purpose of "bringing compute to where data is". This > is wasteful because in most cases code doesn't change much across many jobs. > I'd like to propose and discuss feasibility of introducing a truly shared > cache so that multiple jobs from multiple users can share and cache jars. > This JIRA is to open the discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2183) Cleaner service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184512#comment-14184512 ] Hudson commented on YARN-2183: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/]) YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) (kasha: rev c51e53d7aad46059f52d4046a5fedfdfd3c37955) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/TestCleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/TestCleanerTask.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/metrics/CleanerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java > Cleaner service for cache manager > - > > Key: YARN-2183 > URL: https://issues.apache.org/jira/browse/YARN-2183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2183-trunk-v1.patch, YARN-2183-trunk-v2.patch, > YARN-2183-trunk-v3.patch, YARN-2183-trunk-v4.patch, YARN-2183-trunk-v5.patch, > YARN-2183-trunk-v6.patch, YARN-2183-trunk-v7.patch, YARN-2183-trunk-v8.patch, > on-demand-cleaner.patch > > > Implement the cleaner service for the cache manager along with metrics for > the service. This service is responsible for cleaning up old resource > references in the manager and removing stale entries from the cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2657) MiniYARNCluster to (optionally) add MicroZookeeper service
[
https://issues.apache.org/jira/browse/YARN-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-2657:
-
Attachment: YARN-2657-002.patch
This patch is in sync with the YARN-2571 patch 005; the RM-side root path
creation is now async, so the test has to {{get()}} the future of the async
operation before examining the path.
For that reason, it won't compile until that patch is in
> MiniYARNCluster to (optionally) add MicroZookeeper service
> --
>
> Key: YARN-2657
> URL: https://issues.apache.org/jira/browse/YARN-2657
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-2567-001.patch, YARN-2657-002.patch
>
>
> This is needed for testing things like YARN-2646: add an option for the
> {{MiniYarnCluster}} to start a {{MicroZookeeperService}}.
> This is just another YARN service to create and track the lifecycle. The
> {{MicroZookeeperService}} publishes its binding information for direct takeup
> by the registry services...this can address in-VM race conditions.
> The default setting for this service is "off"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2669) FairScheduler: queueName shouldn't allow periods the allocation.xml
[
https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184540#comment-14184540
]
bc Wong commented on YARN-2669:
---
To qualify what I wrote:
bq. You'd need some escaping rule, like replacing any naturally occurring
single underscore with two underscores, and then replacing a dot with a single
underscore.
That seems to be out of scope here, and could use more discussion and feedback.
> FairScheduler: queueName shouldn't allow periods the allocation.xml
> ---
>
> Key: YARN-2669
> URL: https://issues.apache.org/jira/browse/YARN-2669
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: YARN-2669-1.patch, YARN-2669-2.patch
>
>
> For an allocation file like:
> {noformat}
>
>
> 4096mb,4vcores
>
>
> {noformat}
> Users may wish to config minResources for a queue with full path "root.q1".
> However, right now, fair scheduler will treat this configureation for the
> queue with full name "root.root.q1". We need to print out a warning msg to
> notify users about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2746) YARNDelegationTokenID misses serializing version from the common abstract ID
Vinod Kumar Vavilapalli created YARN-2746: - Summary: YARNDelegationTokenID misses serializing version from the common abstract ID Key: YARN-2746 URL: https://issues.apache.org/jira/browse/YARN-2746 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.6.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jian He I found this during review of YARN-2743. bq. AbstractDTId had a version, we dropped that in the protobuf serialization. We should just write it during the serialization and read it back? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2743) Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed
[ https://issues.apache.org/jira/browse/YARN-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184563#comment-14184563 ] Vinod Kumar Vavilapalli commented on YARN-2743: --- Okay, the patch looks good to me. File YARN-2746 for the pending item. Can you please validate if the test-failures are related or not? > Yarn jobs via oozie fail with failed to renew token (secure) or digest > mismatch (unsecure) errors when RM is being killed > - > > Key: YARN-2743 > URL: https://issues.apache.org/jira/browse/YARN-2743 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Arpit Gupta >Assignee: Jian He >Priority: Blocker > Attachments: YARN-2743.1.patch, YARN-2743.2.patch > > > During our HA testing we have seen yarn jobs run via oozie fail with failed > to renew delegation token errors on secure clusters and digest mismatch > errors on un secure clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2743) Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed
[ https://issues.apache.org/jira/browse/YARN-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184578#comment-14184578 ] Vinod Kumar Vavilapalli commented on YARN-2743: --- Never mind, I analyzed the test failures myself. Two tests are failing - TestAggregatedLogFormat passes for me locally. Cannot find logs on Jenkins. Will follow up separately. - TestFairScheduler.testContinuousScheduling failure is tracked at YARN-2666. > Yarn jobs via oozie fail with failed to renew token (secure) or digest > mismatch (unsecure) errors when RM is being killed > - > > Key: YARN-2743 > URL: https://issues.apache.org/jira/browse/YARN-2743 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Arpit Gupta >Assignee: Jian He >Priority: Blocker > Attachments: YARN-2743.1.patch, YARN-2743.2.patch > > > During our HA testing we have seen yarn jobs run via oozie fail with failed > to renew delegation token errors on secure clusters and digest mismatch > errors on un secure clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2743) Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed
[ https://issues.apache.org/jira/browse/YARN-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184582#comment-14184582 ] Hudson commented on YARN-2743: -- FAILURE: Integrated in Hadoop-trunk-Commit #6347 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6347/]) YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken identifiers to be tampered and thus causing app submission failures in secure mode. Contributed by Jian He. (vinodkv: rev 018664550507981297fd9f91e29408e6b7801ea9) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/RMDelegationTokenIdentifierForTest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/proto/yarn_server_resourcemanager_recovery.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/RMDelegationTokenIdentifierData.java > Yarn jobs via oozie fail with failed to renew token (secure) or digest > mismatch (unsecure) errors when RM is being killed > - > > Key: YARN-2743 > URL: https://issues.apache.org/jira/browse/YARN-2743 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Arpit Gupta >Assignee: Jian He >Priority: Blocker > Attachments: YARN-2743.1.patch, YARN-2743.2.patch > > > During our HA testing we have seen yarn jobs run via oozie fail with failed > to renew delegation token errors on secure clusters and digest mismatch > errors on un secure clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2712) Adding tests about FSQueue and headroom of FairScheduler to TestWorkPreservingRMRestart
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184603#comment-14184603
]
Tsuyoshi OZAWA commented on YARN-2712:
--
[~kkambatl], could you take a look?
> Adding tests about FSQueue and headroom of FairScheduler to
> TestWorkPreservingRMRestart
> ---
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184604#comment-14184604
]
Zhijie Shen commented on YARN-2734:
---
LGTM overall, one nit:
Can we not say "Log file: " here? It's a bit ambiguous to say a file is a dir?
{code}
+ LOG.warn("Log file: " + logFile.getAbsolutePath()
{code}
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-2734:
Attachment: YARN-2734.2.patch
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch, YARN-2734.2.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184610#comment-14184610
]
Xuan Gong commented on YARN-2734:
-
Thanks for view. [~wangzj] [~vvasudev].
Uploaded a new patch to address latest comment
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch, YARN-2734.2.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2747) TestAggregatedLogFormat fails in trunk
Xuan Gong created YARN-2747: --- Summary: TestAggregatedLogFormat fails in trunk Key: YARN-2747 URL: https://issues.apache.org/jira/browse/YARN-2747 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) Time elapsed: 0.047 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-2747: --- Assignee: Xuan Gong > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2743) Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed
[ https://issues.apache.org/jira/browse/YARN-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184621#comment-14184621 ] Xuan Gong commented on YARN-2743: - [~vinodkv] [~jianhe] Test case failure:TestAggregatedLogFormat will be tracked in YARN-2747. > Yarn jobs via oozie fail with failed to renew token (secure) or digest > mismatch (unsecure) errors when RM is being killed > - > > Key: YARN-2743 > URL: https://issues.apache.org/jira/browse/YARN-2743 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Arpit Gupta >Assignee: Jian He >Priority: Blocker > Fix For: 2.6.0 > > Attachments: YARN-2743.1.patch, YARN-2743.2.patch > > > During our HA testing we have seen yarn jobs run via oozie fail with failed > to renew delegation token errors on secure clusters and digest mismatch > errors on un secure clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184622#comment-14184622 ] Xuan Gong commented on YARN-2747: - This testcase testContainerLogsFileAccess in TestAggregatedLogFormat is using NativeIO. That is why we could not re-produce this testcase failure in MacOS (it will ignore it). But I can reproduce it in Centos. Anyway, the reason for this testcase failure is because after YARN-2724, if we could not open the fileInputStream, no matter what is the reason (permission or invalid user), we will just log the error message locally (Originally, we also log the error message into LogFile. That is why this testcase fails). > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184634#comment-14184634
]
Hadoop QA commented on YARN-2734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677199/YARN-2734.2.patch
against trunk revision 9ada928.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5571//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5571//console
This message is automatically generated.
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch, YARN-2734.2.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2747: Attachment: YARN-2747.1.patch > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-2747.1.patch > > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184636#comment-14184636 ] Xuan Gong commented on YARN-2747: - In YARN-2727, by design, we just need to log the error (can not open FileInputStream) locally. > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-2747.1.patch > > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184637#comment-14184637 ] Xuan Gong commented on YARN-2747: - Sorry, should be In YARN-2724 > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-2747.1.patch > > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184646#comment-14184646
]
Zhijie Shen commented on YARN-2734:
---
+1. Will commit the patch
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch, YARN-2734.2.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2734) If a sub-folder is encountered by log aggregator it results in invalid aggregated file
[
https://issues.apache.org/jira/browse/YARN-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184653#comment-14184653
]
Hudson commented on YARN-2734:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6349 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6349/])
YARN-2734. Skipped sub-folders in the local log dir when aggregating logs.
Contributed by Xuan Gong. (zjshen: rev caecd9fffe7c6216be31f3ab65349182045451fa)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
> If a sub-folder is encountered by log aggregator it results in invalid
> aggregated file
> --
>
> Key: YARN-2734
> URL: https://issues.apache.org/jira/browse/YARN-2734
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.5.1
>Reporter: Sumit Mohanty
>Assignee: Xuan Gong
> Fix For: 2.6.0
>
> Attachments: YARN-2734.1.patch, YARN-2734.2.patch
>
>
> See YARN-2724 for some more context on how the error surfaces during "yarn
> logs" call.
> If aggregator sees a sub-folder today it results in the following error when
> reading the logs:
> {noformat}
> Container: container_1413512973198_0019_01_02 on
> c6401.ambari.apache.org_45454
>
> LogType: cmd_data
> LogLength: 4096
> Log Contents:
> Error aggregating log file. Log file :
> /hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data/hadoop/yarn/log/application_1413512973198_0019/container_1413512973198_0019_01_02/cmd_data
> (Is a directory)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2748) Upload logs in the sub-folders under the local log dir when aggregating logs
Zhijie Shen created YARN-2748: - Summary: Upload logs in the sub-folders under the local log dir when aggregating logs Key: YARN-2748 URL: https://issues.apache.org/jira/browse/YARN-2748 Project: Hadoop YARN Issue Type: Sub-task Components: log-aggregation Affects Versions: 2.6.0 Reporter: Zhijie Shen YARN-2734 has a temporal fix to skip sub folders to avoid exception. Ideally, if the app is creating a sub folder and putting its rolling logs there, we need to upload these logs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184656#comment-14184656
]
Hadoop QA commented on YARN-2747:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677200/YARN-2747.1.patch
against trunk revision 9ada928.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5572//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5572//console
This message is automatically generated.
> TestAggregatedLogFormat fails in trunk
> --
>
> Key: YARN-2747
> URL: https://issues.apache.org/jira/browse/YARN-2747
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-2747.1.patch
>
>
> Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<<
> FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
> testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat)
> Time elapsed: 0.047 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184673#comment-14184673
]
Zhijie Shen commented on YARN-2747:
---
Can you check whether TestLogAggregationService is failing for the same reason?
{code}
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1365)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithInterval(TestLogAggregationService.java:1285)
FAILED:
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:355)
FAILED:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290)
FAILED:
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:355)
FAILED:
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:355)
FAILED:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290)
FAILED:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithInterval
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1365)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithInterval(TestLogAggregationService.java:1285)
{code}
> TestAggregatedLogFormat fails in trunk
> --
>
> Key: YARN-2747
> URL: https://issues.apache.org/jira/browse/YARN-2747
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-2747.1.patch
>
>
> Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Tim
[jira] [Updated] (YARN-2505) Support get/add/remove/change labels in RM REST API
[ https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2505: -- Attachment: YARN-2505.9.patch Retrigger build, the warnings don't seem to come from the new code, verifying. > Support get/add/remove/change labels in RM REST API > --- > > Key: YARN-2505 > URL: https://issues.apache.org/jira/browse/YARN-2505 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Craig Welch > Attachments: YARN-2505.1.patch, YARN-2505.3.patch, YARN-2505.4.patch, > YARN-2505.5.patch, YARN-2505.6.patch, YARN-2505.7.patch, YARN-2505.8.patch, > YARN-2505.9.patch, YARN-2505.9.patch, YARN-2505.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2711) TestDefaultContainerExecutor#testContainerLaunchError fails on Windows
[
https://issues.apache.org/jira/browse/YARN-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184677#comment-14184677
]
Craig Welch commented on YARN-2711:
---
+1 looks good to me
> TestDefaultContainerExecutor#testContainerLaunchError fails on Windows
> --
>
> Key: YARN-2711
> URL: https://issues.apache.org/jira/browse/YARN-2711
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2711.0.patch
>
>
> The testContainerLaunchError test fails on Windows with the following error -
> {noformat}
> java.io.FileNotFoundException: File file:/bin/echo does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:737)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:514)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:111)
> at org.apache.hadoop.fs.FilterFs.getFileStatus(FilterFs.java:120)
> at org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1117)
> at org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1113)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1113)
> at org.apache.hadoop.fs.FileContext$Util.copy(FileContext.java:2019)
> at org.apache.hadoop.fs.FileContext$Util.copy(FileContext.java:1978)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:145)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor.testContainerLaunchError(TestDefaultContainerExecutor.java:289)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2712) Adding tests about FSQueue and headroom of FairScheduler to TestWorkPreservingRMRestart
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184679#comment-14184679
]
Tsuyoshi OZAWA commented on YARN-2712:
--
[~adhoot] [~kkambatl], oops, I misread the previous review comment from
Karthik. Thanks for your review, Anubhav.
> Adding tests about FSQueue and headroom of FairScheduler to
> TestWorkPreservingRMRestart
> ---
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2701) Potential race condition in startLocalizer when using LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184687#comment-14184687
]
Xuan Gong commented on YARN-2701:
-
[~aw] Do you have any other comments ?
> Potential race condition in startLocalizer when using LinuxContainerExecutor
> --
>
> Key: YARN-2701
> URL: https://issues.apache.org/jira/browse/YARN-2701
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2701.1.patch, YARN-2701.2.patch, YARN-2701.3.patch,
> YARN-2701.4.patch, YARN-2701.5.patch, YARN-2701.6.patch,
> YARN-2701.addendum.1.patch, YARN-2701.addendum.2.patch,
> YARN-2701.addendum.3.patch, YARN-2701.addendum.4.patch
>
>
> When using LinuxContainerExecutor do startLocalizer, we are using native code
> container-executor.c.
> {code}
> if (stat(npath, &sb) != 0) {
>if (mkdir(npath, perm) != 0) {
> {code}
> We are using check and create method to create the appDir under /usercache.
> But if there are two containers trying to do this at the same time, race
> condition may happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2505) Support get/add/remove/change labels in RM REST API
[
https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184693#comment-14184693
]
Hadoop QA commented on YARN-2505:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677208/YARN-2505.9.patch
against trunk revision caecd9f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5573//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5573//console
This message is automatically generated.
> Support get/add/remove/change labels in RM REST API
> ---
>
> Key: YARN-2505
> URL: https://issues.apache.org/jira/browse/YARN-2505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Craig Welch
> Attachments: YARN-2505.1.patch, YARN-2505.3.patch, YARN-2505.4.patch,
> YARN-2505.5.patch, YARN-2505.6.patch, YARN-2505.7.patch, YARN-2505.8.patch,
> YARN-2505.9.patch, YARN-2505.9.patch, YARN-2505.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2704) Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time
[ https://issues.apache.org/jira/browse/YARN-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2704: -- Attachment: YARN-2704.3.patch bq. It’s inside the run method, so I wrap it with runTimeException Ignore this comment, I changed to use PrivilegedExceptionAction whose run method declares exception. > Localization and log-aggregation will fail if hdfs delegation token expired > after token-max-life-time > -- > > Key: YARN-2704 > URL: https://issues.apache.org/jira/browse/YARN-2704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-2704.1.patch, YARN-2704.2.patch, YARN-2704.2.patch, > YARN-2704.3.patch > > > In secure mode, YARN requires the hdfs-delegation token to do localization > and log aggregation on behalf of the user. But the hdfs delegation token will > eventually expire after max-token-life-time. So, localization and log > aggregation will fail after the token expires. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2704) Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time
[
https://issues.apache.org/jira/browse/YARN-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184770#comment-14184770
]
Hadoop QA commented on YARN-2704:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677214/YARN-2704.3.patch
against trunk revision caecd9f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5574//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5574//console
This message is automatically generated.
> Localization and log-aggregation will fail if hdfs delegation token expired
> after token-max-life-time
> --
>
> Key: YARN-2704
> URL: https://issues.apache.org/jira/browse/YARN-2704
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-2704.1.patch, YARN-2704.2.patch, YARN-2704.2.patch,
> YARN-2704.3.patch
>
>
> In secure mode, YARN requires the hdfs-delegation token to do localization
> and log aggregation on behalf of the user. But the hdfs delegation token will
> eventually expire after max-token-life-time. So, localization and log
> aggregation will fail after the token expires.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[ https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184780#comment-14184780 ] Naganarasimha G R commented on YARN-2729: - Hi [~Wangda], bq. And you need make checkAndThrowLabelName in CommonsNodeLabelsManager public, check the labels if valid before send to RM. Two concerns if we do this: 1. Two times labels will get validated, once in NM and once in RM(NodeLabelsManager.replaceNodeLabels ). And anyway as the exception message is planned to be propagated till NM and getting logged in NM, would this be required.? 2. Currently checkAndThrowLabelName is validating and throwing exception for a singe label, i think it would better to accept a collection and show that which all labels in the collection are not valid. So that user need not wait till multiple timer runs to identify all the set of valid labels. so was planning to rename the method to $$ validateLabels(Collection labels) $$ and make it static to get it accessible for (conf & script)NodeLabelsProvider (if this basic validation is required in NM too) > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup > --- > > Key: YARN-2729 > URL: https://issues.apache.org/jira/browse/YARN-2729 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch > > > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2647) Add yarn queue CLI to get queue infos
[ https://issues.apache.org/jira/browse/YARN-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2647: -- Attachment: 0006-YARN-2647.patch Updated patch as per the comments. Thank you [~wangda] for the review. > Add yarn queue CLI to get queue infos > - > > Key: YARN-2647 > URL: https://issues.apache.org/jira/browse/YARN-2647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: 0001-YARN-2647.patch, 0002-YARN-2647.patch, > 0003-YARN-2647.patch, 0004-YARN-2647.patch, 0005-YARN-2647.patch, > 0006-YARN-2647.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2749) Some testcases from TestLogAggregationService fails in trunk
Xuan Gong created YARN-2749: --- Summary: Some testcases from TestLogAggregationService fails in trunk Key: YARN-2749 URL: https://issues.apache.org/jira/browse/YARN-2749 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Some testcases from TestLogAggregationService fails in trunk. Those can be reproduced in centos Stack Trace: java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290) Stack Trace: java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2747) TestAggregatedLogFormat fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184855#comment-14184855 ] Xuan Gong commented on YARN-2747: - Those are different. NativeIO is enabled only when security is enabled. And in this testcase: testContainerLogsFileAccess, we enable the security. For other testcase failures: testLogAggregationServiceWithInterval/testLogAggregationServiceWithRetention, they failed because of race condition. (create YARN-2749 to track this issue) > TestAggregatedLogFormat fails in trunk > -- > > Key: YARN-2747 > URL: https://issues.apache.org/jira/browse/YARN-2747 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-2747.1.patch > > > Running org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.105 sec <<< > FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat > testContainerLogsFileAccess(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat) > Time elapsed: 0.047 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testContainerLogsFileAccess(TestAggregatedLogFormat.java:346) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2647) Add yarn queue CLI to get queue infos
[
https://issues.apache.org/jira/browse/YARN-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184856#comment-14184856
]
Hadoop QA commented on YARN-2647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677239/0006-YARN-2647.patch
against trunk revision caecd9f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5575//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5575//console
This message is automatically generated.
> Add yarn queue CLI to get queue infos
> -
>
> Key: YARN-2647
> URL: https://issues.apache.org/jira/browse/YARN-2647
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Wangda Tan
>Assignee: Sunil G
> Attachments: 0001-YARN-2647.patch, 0002-YARN-2647.patch,
> 0003-YARN-2647.patch, 0004-YARN-2647.patch, 0005-YARN-2647.patch,
> 0006-YARN-2647.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2647) Add yarn queue CLI to get queue infos
[ https://issues.apache.org/jira/browse/YARN-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184870#comment-14184870 ] Sunil G commented on YARN-2647: --- testResourceTrackerOnHA is not caused by this fix, seems connection exception from registerNodeManager. > Add yarn queue CLI to get queue infos > - > > Key: YARN-2647 > URL: https://issues.apache.org/jira/browse/YARN-2647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: 0001-YARN-2647.patch, 0002-YARN-2647.patch, > 0003-YARN-2647.patch, 0004-YARN-2647.patch, 0005-YARN-2647.patch, > 0006-YARN-2647.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2647) Add yarn queue CLI to get queue infos
[ https://issues.apache.org/jira/browse/YARN-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184892#comment-14184892 ] Wangda Tan commented on YARN-2647: -- Hi [~sunilg], Latest patch LGTM, +1. Thanks, Wangda > Add yarn queue CLI to get queue infos > - > > Key: YARN-2647 > URL: https://issues.apache.org/jira/browse/YARN-2647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: 0001-YARN-2647.patch, 0002-YARN-2647.patch, > 0003-YARN-2647.patch, 0004-YARN-2647.patch, 0005-YARN-2647.patch, > 0006-YARN-2647.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
