[jira] [Created] (YARN-888) clean up POM dependencies
Alejandro Abdelnur created YARN-888: --- Summary: clean up POM dependencies Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695649#comment-13695649 ] Alejandro Abdelnur commented on YARN-888: - this JIRA should be follow up by a JIRA that fixes dependencies based on the dependencies:analyze report clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-353) Add Zookeeper-based store implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695656#comment-13695656 ] Karthik Kambatla commented on YARN-353: --- Thanks Bikas. Mostly looks good. Can you address the findbugs and rebase against trunk. Will post a detailed review (couple of nits) on the updated patch. Add Zookeeper-based store implementation for RMStateStore - Key: YARN-353 URL: https://issues.apache.org/jira/browse/YARN-353 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Hitesh Shah Assignee: Bikas Saha Attachments: YARN-353.1.patch Add store that write RM state data to ZK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-883) Expose Fair Scheduler-specific queue metrics
[ https://issues.apache.org/jira/browse/YARN-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695661#comment-13695661 ] Alejandro Abdelnur commented on YARN-883: - +1, LGTM. Are you following up with a JIRA to improve the UI so the collapse tree shows both CPU and MEM? Please do so. Expose Fair Scheduler-specific queue metrics Key: YARN-883 URL: https://issues.apache.org/jira/browse/YARN-883 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-883-1.patch, YARN-883-1.patch, YARN-883.patch When the Fair Scheduler is enabled, QueueMetrics should include fair share, minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-883) Expose Fair Scheduler-specific queue metrics
[ https://issues.apache.org/jira/browse/YARN-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695663#comment-13695663 ] Sandy Ryza commented on YARN-883: - Thanks, just filed YARN-889 Expose Fair Scheduler-specific queue metrics Key: YARN-883 URL: https://issues.apache.org/jira/browse/YARN-883 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-883-1.patch, YARN-883-1.patch, YARN-883.patch When the Fair Scheduler is enabled, QueueMetrics should include fair share, minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-889) Make bars in Fair Scheduler web UI show multiple resources
Sandy Ryza created YARN-889: --- Summary: Make bars in Fair Scheduler web UI show multiple resources Key: YARN-889 URL: https://issues.apache.org/jira/browse/YARN-889 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza The Fair Scheduler web UI contains bars that make it easier to visualize resource usage relative to fair share and capacity. Currently these bars only reflect memory metrics - they should be augmented to display other resources as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-461) Fair scheduler accepts apps with empty string names and queues
[ https://issues.apache.org/jira/browse/YARN-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695675#comment-13695675 ] Sandy Ryza commented on YARN-461: - Per Arun's suggestion, we don't need to make changes to the common code (RMAppManager). We should reject the application inside the Fair Scheduler. Fair scheduler accepts apps with empty string names and queues -- Key: YARN-461 URL: https://issues.apache.org/jira/browse/YARN-461 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-461.patch, YARN-461.patch, YARN-461.patch When an app is submitted with for the name or queue, the RMAppManager passes it on like it does with any other string. Instead it should probably use the default name and queue, as it does when it encounters a null name or queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-814) Difficult to diagnose a failed container launch when error due to invalid environment variable
[ https://issues.apache.org/jira/browse/YARN-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695678#comment-13695678 ] Jian He commented on YARN-814: -- kick the jenkins Difficult to diagnose a failed container launch when error due to invalid environment variable -- Key: YARN-814 URL: https://issues.apache.org/jira/browse/YARN-814 Project: Hadoop YARN Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Jian He Attachments: YARN-814.1.patch, YARN-814.patch The container's launch script sets up environment variables, symlinks etc. If there is any failure when setting up the basic context ( before the actual user's process is launched ), nothing is captured by the NM. This makes it impossible to diagnose the reason for the failure. To reproduce, set an env var where the value contains characters that throw syntax errors in bash. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-461) Fair scheduler accepts apps with empty string names and queues
[ https://issues.apache.org/jira/browse/YARN-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695679#comment-13695679 ] Wei Yan commented on YARN-461: -- Got it. I'll update the patch. Fair scheduler accepts apps with empty string names and queues -- Key: YARN-461 URL: https://issues.apache.org/jira/browse/YARN-461 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-461.patch, YARN-461.patch, YARN-461.patch When an app is submitted with for the name or queue, the RMAppManager passes it on like it does with any other string. Instead it should probably use the default name and queue, as it does when it encounters a null name or queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-883) Expose Fair Scheduler-specific queue metrics
[ https://issues.apache.org/jira/browse/YARN-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695685#comment-13695685 ] Hudson commented on YARN-883: - Integrated in Hadoop-trunk-Commit #4018 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4018/]) YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu) (Revision 1497884) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1497884 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java Expose Fair Scheduler-specific queue metrics Key: YARN-883 URL: https://issues.apache.org/jira/browse/YARN-883 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.2.0 Attachments: YARN-883-1.patch, YARN-883-1.patch, YARN-883.patch When the Fair Scheduler is enabled, QueueMetrics should include fair share, minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-744: --- Summary: Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. (was: Locking not correct in org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(AllocateRequest request)) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. - Key: YARN-744 URL: https://issues.apache.org/jira/browse/YARN-744 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Bikas Saha Assignee: Omkar Vinit Joshi Attachments: MAPREDUCE-3899-branch-0.23.patch Looks like the lock taken in this is broken. It takes a lock on lastResponse object and then puts a new lastResponse object into the map. At this point a new thread entering this function will get a new lastResponse object and will be able to take its lock and enter the critical section. Presumably we want to limit one response per app attempt. So the lock could be taken on the ApplicationAttemptId key of the response map object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695706#comment-13695706 ] Omkar Vinit Joshi commented on YARN-744: The problem here is that we retrieve the last response from resource map and then try to grab a lock on it. However after grabbing lock we don't check if the last response in resource map itself got updated or not. That results into a race condition which I am trying to solve here.. After grabbing the lock an additional check has to be made to ensure that lastResponse was not changed in between i.e. no other AM requests were processed. Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. - Key: YARN-744 URL: https://issues.apache.org/jira/browse/YARN-744 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Bikas Saha Assignee: Omkar Vinit Joshi Attachments: MAPREDUCE-3899-branch-0.23.patch Looks like the lock taken in this is broken. It takes a lock on lastResponse object and then puts a new lastResponse object into the map. At this point a new thread entering this function will get a new lastResponse object and will be able to take its lock and enter the critical section. Presumably we want to limit one response per app attempt. So the lock could be taken on the ApplicationAttemptId key of the response map object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-744: --- Attachment: YARN-744.patch Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. - Key: YARN-744 URL: https://issues.apache.org/jira/browse/YARN-744 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Bikas Saha Assignee: Omkar Vinit Joshi Attachments: MAPREDUCE-3899-branch-0.23.patch, YARN-744.patch Looks like the lock taken in this is broken. It takes a lock on lastResponse object and then puts a new lastResponse object into the map. At this point a new thread entering this function will get a new lastResponse object and will be able to take its lock and enter the critical section. Presumably we want to limit one response per app attempt. So the lock could be taken on the ApplicationAttemptId key of the response map object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-718) Remove RemoteUGI.getRemoteUser check from startContainer in ContainerManagerImpl as this is no longer required
[ https://issues.apache.org/jira/browse/YARN-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi resolved YARN-718. Resolution: Invalid YARN-613 fixes this.. getUser is required and containerId is replaced by applicationAttemptId. Remove RemoteUGI.getRemoteUser check from startContainer in ContainerManagerImpl as this is no longer required -- Key: YARN-718 URL: https://issues.apache.org/jira/browse/YARN-718 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Earlier there was a check in startContainer which was validating that RemoteUGI.getRemoteUser is same as containerId. However this check is no longer required and it should be removed. YARN-699 and YARN-715 will get fixed with this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[
https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695757#comment-13695757
]
Hadoop QA commented on YARN-744:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590077/YARN-744.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1404//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1404//console
This message is automatically generated.
Race condition in ApplicationMasterService.allocate .. It might process same
allocate request twice resulting in additional containers getting allocated.
-
Key: YARN-744
URL: https://issues.apache.org/jira/browse/YARN-744
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Omkar Vinit Joshi
Attachments: MAPREDUCE-3899-branch-0.23.patch, YARN-744.patch
Looks like the lock taken in this is broken. It takes a lock on lastResponse
object and then puts a new lastResponse object into the map. At this point a
new thread entering this function will get a new lastResponse object and will
be able to take its lock and enter the critical section. Presumably we want
to limit one response per app attempt. So the lock could be taken on the
ApplicationAttemptId key of the response map object.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[
https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695766#comment-13695766
]
Hadoop QA commented on YARN-62:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589212/YARN-62-20130621.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.TestApplication
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1405//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1405//console
This message is automatically generated.
AM should not be able to abuse container tokens for repetitive container
launches
-
Key: YARN-62
URL: https://issues.apache.org/jira/browse/YARN-62
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Vinod Kumar Vavilapalli
Assignee: Omkar Vinit Joshi
Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch
Clone of YARN-51.
ApplicationMaster should not be able to store container tokens and use the
same set of tokens for repetitive container launches. The possibility of such
abuse is there in the current code, for a duration of 1d+10mins, we need to
fix this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-890) The roundup for memory values on resource manager UI is misleading
Trupti Dhavle created YARN-890: -- Summary: The roundup for memory values on resource manager UI is misleading Key: YARN-890 URL: https://issues.apache.org/jira/browse/YARN-890 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Trupti Dhavle From the yarn-site.xml, I see following values- property nameyarn.nodemanager.resource.memory-mb/name value4192/value /property property nameyarn.scheduler.maximum-allocation-mb/name value4192/value /property property nameyarn.scheduler.minimum-allocation-mb/name value1024/value /property However the resourcemanager UI shows total memory as 5MB -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-513) Create common proxy client for communicating with RM
[ https://issues.apache.org/jira/browse/YARN-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-513: - Attachment: YARN-513.13.patch rebased on latest trunk Create common proxy client for communicating with RM Key: YARN-513 URL: https://issues.apache.org/jira/browse/YARN-513 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-513.10.patch, YARN-513.11.patch, YARN-513.12.patch, YARN-513.13.patch, YARN-513.1.patch, YARN-513.2.patch, YARN-513.3.patch, YARN-513.4.patch, YARN.513.5.patch, YARN-513.6.patch, YARN-513.7.patch, YARN-513.8.patch, YARN-513.9.patch When the RM is restarting, the NM, AM and Clients should wait for some time for the RM to come back up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-814) Difficult to diagnose a failed container launch when error due to invalid environment variable
[ https://issues.apache.org/jira/browse/YARN-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-814: - Attachment: YARN-814.2.patch Difficult to diagnose a failed container launch when error due to invalid environment variable -- Key: YARN-814 URL: https://issues.apache.org/jira/browse/YARN-814 Project: Hadoop YARN Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Jian He Attachments: YARN-814.1.patch, YARN-814.2.patch, YARN-814.patch The container's launch script sets up environment variables, symlinks etc. If there is any failure when setting up the basic context ( before the actual user's process is launched ), nothing is captured by the NM. This makes it impossible to diagnose the reason for the failure. To reproduce, set an env var where the value contains characters that throw syntax errors in bash. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-513) Create common proxy client for communicating with RM
[
https://issues.apache.org/jira/browse/YARN-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695859#comment-13695859
]
Hadoop QA commented on YARN-513:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590095/YARN-513.13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1406//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1406//console
This message is automatically generated.
Create common proxy client for communicating with RM
Key: YARN-513
URL: https://issues.apache.org/jira/browse/YARN-513
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-513.10.patch, YARN-513.11.patch, YARN-513.12.patch,
YARN-513.13.patch, YARN-513.1.patch, YARN-513.2.patch, YARN-513.3.patch,
YARN-513.4.patch, YARN.513.5.patch, YARN-513.6.patch, YARN-513.7.patch,
YARN-513.8.patch, YARN-513.9.patch
When the RM is restarting, the NM, AM and Clients should wait for some time
for the RM to come back up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-814) Difficult to diagnose a failed container launch when error due to invalid environment variable
[
https://issues.apache.org/jira/browse/YARN-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695874#comment-13695874
]
Hadoop QA commented on YARN-814:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590098/YARN-814.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1407//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1407//console
This message is automatically generated.
Difficult to diagnose a failed container launch when error due to invalid
environment variable
--
Key: YARN-814
URL: https://issues.apache.org/jira/browse/YARN-814
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Jian He
Attachments: YARN-814.1.patch, YARN-814.2.patch, YARN-814.patch
The container's launch script sets up environment variables, symlinks etc.
If there is any failure when setting up the basic context ( before the actual
user's process is launched ), nothing is captured by the NM. This makes it
impossible to diagnose the reason for the failure.
To reproduce, set an env var where the value contains characters that throw
syntax errors in bash.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[ https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695883#comment-13695883 ] Omkar Vinit Joshi commented on YARN-62: --- fixing test issues... AM should not be able to abuse container tokens for repetitive container launches - Key: YARN-62 URL: https://issues.apache.org/jira/browse/YARN-62 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch Clone of YARN-51. ApplicationMaster should not be able to store container tokens and use the same set of tokens for repetitive container launches. The possibility of such abuse is there in the current code, for a duration of 1d+10mins, we need to fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[ https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-62: -- Attachment: YARN-62-20130628.patch AM should not be able to abuse container tokens for repetitive container launches - Key: YARN-62 URL: https://issues.apache.org/jira/browse/YARN-62 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch, YARN-62-20130628.patch Clone of YARN-51. ApplicationMaster should not be able to store container tokens and use the same set of tokens for repetitive container launches. The possibility of such abuse is there in the current code, for a duration of 1d+10mins, we need to fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[ https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-62: -- Attachment: (was: YARN-62-20130628.patch) AM should not be able to abuse container tokens for repetitive container launches - Key: YARN-62 URL: https://issues.apache.org/jira/browse/YARN-62 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch, YARN-62-20130628.patch Clone of YARN-51. ApplicationMaster should not be able to store container tokens and use the same set of tokens for repetitive container launches. The possibility of such abuse is there in the current code, for a duration of 1d+10mins, we need to fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[ https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-62: -- Attachment: YARN-62-20130628.patch AM should not be able to abuse container tokens for repetitive container launches - Key: YARN-62 URL: https://issues.apache.org/jira/browse/YARN-62 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch, YARN-62-20130628.patch Clone of YARN-51. ApplicationMaster should not be able to store container tokens and use the same set of tokens for repetitive container launches. The possibility of such abuse is there in the current code, for a duration of 1d+10mins, we need to fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-890) The roundup for memory values on resource manager UI is misleading
[ https://issues.apache.org/jira/browse/YARN-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695891#comment-13695891 ] Karthik Kambatla commented on YARN-890: --- Hi Trupti Can you provide more information, may be a screen shot of the resource manager ui you are referring to? The roundup for memory values on resource manager UI is misleading -- Key: YARN-890 URL: https://issues.apache.org/jira/browse/YARN-890 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Trupti Dhavle From the yarn-site.xml, I see following values- property nameyarn.nodemanager.resource.memory-mb/name value4192/value /property property nameyarn.scheduler.maximum-allocation-mb/name value4192/value /property property nameyarn.scheduler.minimum-allocation-mb/name value1024/value /property However the resourcemanager UI shows total memory as 5MB -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-62) AM should not be able to abuse container tokens for repetitive container launches
[
https://issues.apache.org/jira/browse/YARN-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695908#comment-13695908
]
Hadoop QA commented on YARN-62:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590108/YARN-62-20130628.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1408//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1408//console
This message is automatically generated.
AM should not be able to abuse container tokens for repetitive container
launches
-
Key: YARN-62
URL: https://issues.apache.org/jira/browse/YARN-62
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Vinod Kumar Vavilapalli
Assignee: Omkar Vinit Joshi
Attachments: YARN-62-20130621.1.patch, YARN-62-20130621.patch,
YARN-62-20130628.patch
Clone of YARN-51.
ApplicationMaster should not be able to store container tokens and use the
same set of tokens for repetitive container launches. The possibility of such
abuse is there in the current code, for a duration of 1d+10mins, we need to
fix this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-877) Allow for black-listing resources in FifoScheduler
[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-877: Attachment: YARN-877-2.patch Remove some unnecessary changes that fix warnings, we can address them separated later. Allow for black-listing resources in FifoScheduler -- Key: YARN-877 URL: https://issues.apache.org/jira/browse/YARN-877 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Junping Du Assignee: Junping Du Attachments: YARN-877-2.patch, YARN-877.patch YARN-750 already addressed black-list staff in YARN API and CS scheduler, this jira add implementation for FifoScheduler. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler
[
https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695954#comment-13695954
]
Hadoop QA commented on YARN-877:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590115/YARN-877-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1409//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1409//console
This message is automatically generated.
Allow for black-listing resources in FifoScheduler
--
Key: YARN-877
URL: https://issues.apache.org/jira/browse/YARN-877
Project: Hadoop YARN
Issue Type: Sub-task
Components: scheduler
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-877-2.patch, YARN-877.patch
YARN-750 already addressed black-list staff in YARN API and CS scheduler,
this jira add implementation for FifoScheduler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
