[jira] [Commented] (YARN-1774) FS: Submitting to non-leaf queue throws NPE
[ https://issues.apache.org/jira/browse/YARN-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924791#comment-13924791 ] Hudson commented on YARN-1774: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) YARN-1774. FS: Submitting to non-leaf queue throws NPE. (Anubhav Dhoot and Karthik Kambatla via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575415) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java FS: Submitting to non-leaf queue throws NPE --- Key: YARN-1774 URL: https://issues.apache.org/jira/browse/YARN-1774 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1774.patch, yarn-1774-2.patch, yarn-1774-3.patch If you create a hierarchy of queues and assign a job to parent queue, FairScheduler quits with a NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1790) Fair Scheduler UI not showing apps table
[ https://issues.apache.org/jira/browse/YARN-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924793#comment-13924793 ] Hudson commented on YARN-1790: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) YARN-1790. Fair Scheduler UI not showing apps table (bc Wong via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java Fair Scheduler UI not showing apps table Key: YARN-1790 URL: https://issues.apache.org/jira/browse/YARN-1790 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.3.0 Reporter: bc Wong Assignee: bc Wong Fix For: 2.5.0 Attachments: 0001-YARN-1790.-FairScheduler-UI-not-showing-apps-table.patch, fs_ui.png, fs_ui_fixed.png There is a running job, which shows up in the summary table in the FairScheduler UI, the queue display, etc. Just not in the apps table at the bottom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1783) yarn application does not make any progress even when no other application is running when RM is being restarted in the background
[ https://issues.apache.org/jira/browse/YARN-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924799#comment-13924799 ] Hudson commented on YARN-1783: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) YARN-1783. Fixed a bug in NodeManager's status-updater that was losing completed container statuses when NodeManager is forced to resync by the ResourceManager. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575437) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdater.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java yarn application does not make any progress even when no other application is running when RM is being restarted in the background -- Key: YARN-1783 URL: https://issues.apache.org/jira/browse/YARN-1783 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Arpit Gupta Assignee: Jian He Priority: Critical Fix For: 2.4.0 Attachments: YARN-1783.1.patch, YARN-1783.2.patch, YARN-1783.3.patch, YARN-1783.4.patch Noticed that during HA tests some tests took over 3 hours to run when the test failed. Looking at the logs i see the application made no progress for a very long time. However if i look at application log from yarn it actually ran in 5 mins I am seeing same behavior when RM was being restarted in the background and when both RM and AM were being restarted. This does not happen for all applications but a few will hit this in the nightly run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1787) yarn applicationattempt/container print wrong usage information
[
https://issues.apache.org/jira/browse/YARN-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924802#comment-13924802
]
Hudson commented on YARN-1787:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/503/])
YARN-1787. Fixed help messages for applicationattempt and container
sub-commands in bin/yarn. Contributed by Zhijie Shen. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575482)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
yarn applicationattempt/container print wrong usage information
---
Key: YARN-1787
URL: https://issues.apache.org/jira/browse/YARN-1787
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Fix For: 2.4.0
Attachments: ApplicationCLI.java.rej, YARN-1787.1.patch,
YARN-1787.2.patch
yarn applicationattempt prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
yarn container prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
Both commands print irrelevant yarn application usage information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924800#comment-13924800 ] Hudson commented on YARN-1410: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) YARN-1410. Added tests to validate that clients can fail-over to a new RM after getting an application-ID but before submission and can still submit to the newly active RM with no issues. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575478) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNewApplicationResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/RMHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestKillApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java Handle RM fails over after getApplicationID() and before submitApplication(). - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Fix For: 2.4.0 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.10.patch, YARN-1410.10.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, YARN-1410.9.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1781) NM should allow users to specify max disk utilization for local disks
[ https://issues.apache.org/jira/browse/YARN-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924796#comment-13924796 ] Hudson commented on YARN-1781: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) YARN-1781. Modified NodeManagers to allow admins to specify max disk utilization for local disks so as to be able to offline full disks. Contributed by Varun Vasudev. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575463) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java NM should allow users to specify max disk utilization for local disks - Key: YARN-1781 URL: https://issues.apache.org/jira/browse/YARN-1781 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Fix For: 2.4.0 Attachments: apache-yarn-1781.0.patch, apache-yarn-1781.1.patch, apache-yarn-1781.2.patch, apache-yarn-1781.3.patch, apache-yarn-1781.4.patch This is related to YARN-257(it's probably a sub task?). Currently, the NM does not detect full disks and allows full disks to be used by containers leading to repeated failures. YARN-257 deals with graceful handling of full disks. This ticket is only about detection of full disks by the disk health checkers. The NM should allow users to set a maximum disk utilization for local disks and mark disks as bad once they exceed that utilization. At the very least, the NM should at least detect full disks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1790) Fair Scheduler UI not showing apps table
[ https://issues.apache.org/jira/browse/YARN-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924850#comment-13924850 ] Hudson commented on YARN-1790: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/]) YARN-1790. Fair Scheduler UI not showing apps table (bc Wong via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java Fair Scheduler UI not showing apps table Key: YARN-1790 URL: https://issues.apache.org/jira/browse/YARN-1790 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.3.0 Reporter: bc Wong Assignee: bc Wong Fix For: 2.5.0 Attachments: 0001-YARN-1790.-FairScheduler-UI-not-showing-apps-table.patch, fs_ui.png, fs_ui_fixed.png There is a running job, which shows up in the summary table in the FairScheduler UI, the queue display, etc. Just not in the apps table at the bottom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1774) FS: Submitting to non-leaf queue throws NPE
[ https://issues.apache.org/jira/browse/YARN-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924848#comment-13924848 ] Hudson commented on YARN-1774: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/]) YARN-1774. FS: Submitting to non-leaf queue throws NPE. (Anubhav Dhoot and Karthik Kambatla via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575415) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java FS: Submitting to non-leaf queue throws NPE --- Key: YARN-1774 URL: https://issues.apache.org/jira/browse/YARN-1774 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1774.patch, yarn-1774-2.patch, yarn-1774-3.patch If you create a hierarchy of queues and assign a job to parent queue, FairScheduler quits with a NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1783) yarn application does not make any progress even when no other application is running when RM is being restarted in the background
[ https://issues.apache.org/jira/browse/YARN-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924856#comment-13924856 ] Hudson commented on YARN-1783: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/]) YARN-1783. Fixed a bug in NodeManager's status-updater that was losing completed container statuses when NodeManager is forced to resync by the ResourceManager. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575437) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdater.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java yarn application does not make any progress even when no other application is running when RM is being restarted in the background -- Key: YARN-1783 URL: https://issues.apache.org/jira/browse/YARN-1783 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Arpit Gupta Assignee: Jian He Priority: Critical Fix For: 2.4.0 Attachments: YARN-1783.1.patch, YARN-1783.2.patch, YARN-1783.3.patch, YARN-1783.4.patch Noticed that during HA tests some tests took over 3 hours to run when the test failed. Looking at the logs i see the application made no progress for a very long time. However if i look at application log from yarn it actually ran in 5 mins I am seeing same behavior when RM was being restarted in the background and when both RM and AM were being restarted. This does not happen for all applications but a few will hit this in the nightly run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1787) yarn applicationattempt/container print wrong usage information
[
https://issues.apache.org/jira/browse/YARN-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924859#comment-13924859
]
Hudson commented on YARN-1787:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/])
YARN-1787. Fixed help messages for applicationattempt and container
sub-commands in bin/yarn. Contributed by Zhijie Shen. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575482)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
yarn applicationattempt/container print wrong usage information
---
Key: YARN-1787
URL: https://issues.apache.org/jira/browse/YARN-1787
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Fix For: 2.4.0
Attachments: ApplicationCLI.java.rej, YARN-1787.1.patch,
YARN-1787.2.patch
yarn applicationattempt prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
yarn container prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
Both commands print irrelevant yarn application usage information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924857#comment-13924857 ] Hudson commented on YARN-1410: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/]) YARN-1410. Added tests to validate that clients can fail-over to a new RM after getting an application-ID but before submission and can still submit to the newly active RM with no issues. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575478) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNewApplicationResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/RMHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestKillApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java Handle RM fails over after getApplicationID() and before submitApplication(). - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Fix For: 2.4.0 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.10.patch, YARN-1410.10.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, YARN-1410.9.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1774) FS: Submitting to non-leaf queue throws NPE
[ https://issues.apache.org/jira/browse/YARN-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924875#comment-13924875 ] Hudson commented on YARN-1774: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/]) YARN-1774. FS: Submitting to non-leaf queue throws NPE. (Anubhav Dhoot and Karthik Kambatla via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575415) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java FS: Submitting to non-leaf queue throws NPE --- Key: YARN-1774 URL: https://issues.apache.org/jira/browse/YARN-1774 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1774.patch, yarn-1774-2.patch, yarn-1774-3.patch If you create a hierarchy of queues and assign a job to parent queue, FairScheduler quits with a NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1781) NM should allow users to specify max disk utilization for local disks
[ https://issues.apache.org/jira/browse/YARN-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924880#comment-13924880 ] Hudson commented on YARN-1781: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/]) YARN-1781. Modified NodeManagers to allow admins to specify max disk utilization for local disks so as to be able to offline full disks. Contributed by Varun Vasudev. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575463) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java NM should allow users to specify max disk utilization for local disks - Key: YARN-1781 URL: https://issues.apache.org/jira/browse/YARN-1781 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Fix For: 2.4.0 Attachments: apache-yarn-1781.0.patch, apache-yarn-1781.1.patch, apache-yarn-1781.2.patch, apache-yarn-1781.3.patch, apache-yarn-1781.4.patch This is related to YARN-257(it's probably a sub task?). Currently, the NM does not detect full disks and allows full disks to be used by containers leading to repeated failures. YARN-257 deals with graceful handling of full disks. This ticket is only about detection of full disks by the disk health checkers. The NM should allow users to set a maximum disk utilization for local disks and mark disks as bad once they exceed that utilization. At the very least, the NM should at least detect full disks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924884#comment-13924884 ] Hudson commented on YARN-1410: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/]) YARN-1410. Added tests to validate that clients can fail-over to a new RM after getting an application-ID but before submission and can still submit to the newly active RM with no issues. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575478) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNewApplicationResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/RMHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestKillApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java Handle RM fails over after getApplicationID() and before submitApplication(). - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Fix For: 2.4.0 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.10.patch, YARN-1410.10.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, YARN-1410.9.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1790) Fair Scheduler UI not showing apps table
[ https://issues.apache.org/jira/browse/YARN-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924877#comment-13924877 ] Hudson commented on YARN-1790: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/]) YARN-1790. Fair Scheduler UI not showing apps table (bc Wong via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java Fair Scheduler UI not showing apps table Key: YARN-1790 URL: https://issues.apache.org/jira/browse/YARN-1790 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.3.0 Reporter: bc Wong Assignee: bc Wong Fix For: 2.5.0 Attachments: 0001-YARN-1790.-FairScheduler-UI-not-showing-apps-table.patch, fs_ui.png, fs_ui_fixed.png There is a running job, which shows up in the summary table in the FairScheduler UI, the queue display, etc. Just not in the apps table at the bottom. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1787) yarn applicationattempt/container print wrong usage information
[
https://issues.apache.org/jira/browse/YARN-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924886#comment-13924886
]
Hudson commented on YARN-1787:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/])
YARN-1787. Fixed help messages for applicationattempt and container
sub-commands in bin/yarn. Contributed by Zhijie Shen. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1575482)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
yarn applicationattempt/container print wrong usage information
---
Key: YARN-1787
URL: https://issues.apache.org/jira/browse/YARN-1787
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Fix For: 2.4.0
Attachments: ApplicationCLI.java.rej, YARN-1787.1.patch,
YARN-1787.2.patch
yarn applicationattempt prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
yarn container prints:
{code}
Invalid Command Usage :
usage: application
-appStates States Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Types Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Application ID Kills the application.
-list arg List application attempts for aplication
from AHS.
-movetoqueue Application ID Moves the application to a different
queue.
-queue Queue Name Works with the movetoqueue command to
specify which queue to move an
application to.
-status Application IDPrints the status of the application.
{code}
Both commands print irrelevant yarn application usage information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (YARN-1804) Signal container request delivery from client to resourcemanager
[ https://issues.apache.org/jira/browse/YARN-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-1804: --- Assignee: Xuan Gong Signal container request delivery from client to resourcemanager Key: YARN-1804 URL: https://issues.apache.org/jira/browse/YARN-1804 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Reporter: Ming Ma Assignee: Xuan Gong It could include the following work items 1. Define the OS independent SignalContainerCMD enum commands. We will start with known requirements such as QUIT. We can expand the list later. 2. Add a new method signalContainer to ApplicationClientProtocol. signalContainerRequest will include containerId as well as SignalContainerCMD. 3. Add signalContainer method to YarnClient and YarnClientImpl. 4. RM will deliver the request to the RMNode object that owns the container. 5. RM needs to have the proper authorization for the signal request. -- This message was sent by Atlassian JIRA (v6.2#6252)
