[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747319#comment-13747319 ] Mohammad Kamrul Islam commented on YARN-49: --- [~ojoshi] do you have WIP patch that i can use for new Giraph AM? It doesn't need to work though. Improve distributed shell application to work on a secure cluster - Key: YARN-49 URL: https://issues.apache.org/jira/browse/YARN-49 Project: Hadoop YARN Issue Type: Sub-task Components: applications/distributed-shell Reporter: Hitesh Shah Assignee: Omkar Vinit Joshi -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1091) All containers localization fails in NM when any one of the configured nm local-dir disk becomes full
[
https://issues.apache.org/jira/browse/YARN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated YARN-1091:
Summary: All containers localization fails in NM when any one of the
configured nm local-dir disk becomes full (was: All containers localization
fails when any one of the configured nm local-dir disk becomes full)
All containers localization fails in NM when any one of the configured nm
local-dir disk becomes full
-
Key: YARN-1091
URL: https://issues.apache.org/jira/browse/YARN-1091
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
{code:xml}
2013-08-22 13:54:22,100 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
create app directory
/opt/nish/usercache/nish/appcache/application_1377151891396_0017
java.io.IOException: mkdir of
/opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
from
/home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
=
file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
2013-08-22 13:54:22,103 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.FileNotFoundException: File
file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does
not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,104 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1377151891396_0017_01_000263 transitioned from
LOCALIZING to LOCALIZATION_FAILED
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1091) All containers localization fails when any one of the configured nm local-dir disk becomes full
Devaraj K created YARN-1091:
---
Summary: All containers localization fails when any one of the
configured nm local-dir disk becomes full
Key: YARN-1091
URL: https://issues.apache.org/jira/browse/YARN-1091
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
{code:xml}
2013-08-22 13:54:22,100 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
create app directory
/opt/nish/usercache/nish/appcache/application_1377151891396_0017
java.io.IOException: mkdir of
/opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
from
/home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
=
file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
2013-08-22 13:54:22,103 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.FileNotFoundException: File
file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,104 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1377151891396_0017_01_000263 transitioned from LOCALIZING
to LOCALIZATION_FAILED
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747422#comment-13747422 ] Hudson commented on YARN-1082: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #309 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/309/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java Secure RM with recovery enabled and rm state store on hdfs fails with gss exception --- Key: YARN-1082 URL: https://issues.apache.org/jira/browse/YARN-1082 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747426#comment-13747426 ] Hudson commented on YARN-881: - SUCCESS: Integrated in Hadoop-Yarn-trunk #309 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/309/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java Priority#compareTo method seems to be wrong. Key: YARN-881 URL: https://issues.apache.org/jira/browse/YARN-881 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-881.1.patch, YARN-881.patch if lower int value means higher priority, shouldn't we return other.getPriority() - this.getPriority() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747516#comment-13747516 ] Hudson commented on YARN-881: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1499 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1499/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java Priority#compareTo method seems to be wrong. Key: YARN-881 URL: https://issues.apache.org/jira/browse/YARN-881 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-881.1.patch, YARN-881.patch if lower int value means higher priority, shouldn't we return other.getPriority() - this.getPriority() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747512#comment-13747512 ] Hudson commented on YARN-1082: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1499 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1499/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java Secure RM with recovery enabled and rm state store on hdfs fails with gss exception --- Key: YARN-1082 URL: https://issues.apache.org/jira/browse/YARN-1082 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747543#comment-13747543 ] Hudson commented on YARN-881: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1526 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1526/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java Priority#compareTo method seems to be wrong. Key: YARN-881 URL: https://issues.apache.org/jira/browse/YARN-881 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-881.1.patch, YARN-881.patch if lower int value means higher priority, shouldn't we return other.getPriority() - this.getPriority() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747539#comment-13747539 ] Hudson commented on YARN-1082: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1526 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1526/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java Secure RM with recovery enabled and rm state store on hdfs fails with gss exception --- Key: YARN-1082 URL: https://issues.apache.org/jira/browse/YARN-1082 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1091) All containers localization fails in NM when any one of the configured nm local-dir disk becomes full
[
https://issues.apache.org/jira/browse/YARN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-1091.
--
Resolution: Duplicate
Duplicate of YARN-257.
All containers localization fails in NM when any one of the configured nm
local-dir disk becomes full
-
Key: YARN-1091
URL: https://issues.apache.org/jira/browse/YARN-1091
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
{code:xml}
2013-08-22 13:54:22,100 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
create app directory
/opt/nish/usercache/nish/appcache/application_1377151891396_0017
java.io.IOException: mkdir of
/opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
from
/home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
=
file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
2013-08-22 13:54:22,103 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.FileNotFoundException: File
file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does
not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,104 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1377151891396_0017_01_000263 transitioned from
LOCALIZING to LOCALIZATION_FAILED
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747701#comment-13747701 ] Sandy Ryza commented on YARN-1024: -- Filed YARN-1089 for adding YCUs. Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1089: - Description: Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. Add YARN compute units alongside virtual cores -- Key: YARN-1089 URL: https://issues.apache.org/jira/browse/YARN-1089 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-899) Get queue administration ACLs working
[ https://issues.apache.org/jira/browse/YARN-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-899: --- Attachment: YARN-899.6.patch create the patch based on the latest trunk Get queue administration ACLs working - Key: YARN-899 URL: https://issues.apache.org/jira/browse/YARN-899 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Xuan Gong Attachments: YARN-899.1.patch, YARN-899.2.patch, YARN-899.3.patch, YARN-899.4.patch, YARN-899.5.patch, YARN-899.5.patch, YARN-899.6.patch The Capacity Scheduler documents the yarn.scheduler.capacity.root.queue-path.acl_administer_queue config option for controlling who can administer a queue, but it is not hooked up to anything. The Fair Scheduler could make use of a similar option as well. This is a feature-parity regression from MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747750#comment-13747750 ] Arun C Murthy commented on YARN-1089: - +1 for this enhancement. Add YARN compute units alongside virtual cores -- Key: YARN-1089 URL: https://issues.apache.org/jira/browse/YARN-1089 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747851#comment-13747851 ] Vinod Kumar Vavilapalli commented on YARN-1089: --- I haven't been following YARN-1024, can you please summarize the proposal here for discussion? Thanks. Add YARN compute units alongside virtual cores -- Key: YARN-1089 URL: https://issues.apache.org/jira/browse/YARN-1089 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747855#comment-13747855 ] Hitesh Shah commented on YARN-1089: --- +1 to [~vinodkv]'s request. [~sandyr] Could you ensure that the proposal also clearly explains how an application developer is meant to use the compute units and/or virtual cores when defining an allocation request and how the allocation (based on these 2 params) will be enforced on a container. Add YARN compute units alongside virtual cores -- Key: YARN-1089 URL: https://issues.apache.org/jira/browse/YARN-1089 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747881#comment-13747881 ] Sandy Ryza commented on YARN-1089: -- Yeah, I'll write up a document and post it on YARN-1024. I'm hoping to keep the broader discussion there so we can use this (and perhaps additional JIRAs) for the actual implementation. Add YARN compute units alongside virtual cores -- Key: YARN-1089 URL: https://issues.apache.org/jira/browse/YARN-1089 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1092) RM proxy URLs invalid when running without a YARN proxy service
Steve Loughran created YARN-1092: Summary: RM proxy URLs invalid when running without a YARN proxy service Key: YARN-1092 URL: https://issues.apache.org/jira/browse/YARN-1092 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Environment: single host cluster without any yarn proxy service Reporter: Steve Loughran Priority: Minor The URLs generated in the RM GUI aren't valid if you don't have a proxy server running -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1092) RM proxy URLs invalid when running without a YARN proxy service
[
https://issues.apache.org/jira/browse/YARN-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747896#comment-13747896
]
Steve Loughran commented on YARN-1092:
--
# My AM generates a URL to the hbase master, such as {{http://ubuntu:8080}}, a
URL registered with an AM that is not running a proxy service, and whose
yarn-site.xml does not declare one
# the link on the GUI is {{http://ubuntu:9081//ubuntu:8080}}
# Which leads to a 404 page
RM proxy URLs invalid when running without a YARN proxy service
---
Key: YARN-1092
URL: https://issues.apache.org/jira/browse/YARN-1092
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Environment: single host cluster without any yarn proxy service
Reporter: Steve Loughran
Priority: Minor
The URLs generated in the RM GUI aren't valid if you don't have a proxy
server running
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-540) RM state store not cleaned if job succeeds but RM shutdown and restart-dispatcher stopped before it can process REMOVE_APP event
[ https://issues.apache.org/jira/browse/YARN-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748024#comment-13748024 ] Vinod Kumar Vavilapalli commented on YARN-540: -- I think we should fix it the right way. And after things like changes in MAPREDUCE-5476, AM's spending time in FINISHING state is going to be more common. So, I am +1 to Bikas's proposal (2). But in the interesting of not making incompatible changes, let's do the following: - Let's change FinishApplicationMasterResponse to also container a response-completed field. If it is true, it means that RM has finished the finalization of AM, otherwise, AM is supposed to retry till it becomes true - Let RM do the state-store changes asynchronously. It'll still be a behavior change, but clients which don't follow the multi-step unregister will risk only getting restarted. RM state store not cleaned if job succeeds but RM shutdown and restart-dispatcher stopped before it can process REMOVE_APP event Key: YARN-540 URL: https://issues.apache.org/jira/browse/YARN-540 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Jian He Priority: Blocker Attachments: YARN-540.patch, YARN-540.patch When job succeeds and successfully call finishApplicationMaster, RM shutdown and restart-dispatcher is stopped before it can process REMOVE_APP event. The next time RM comes back, it will reload the existing state files even though the job is succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-540) Race condition causing RM to potentially relaunch already unregistered AMs on RM restart
[ https://issues.apache.org/jira/browse/YARN-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-540: - Summary: Race condition causing RM to potentially relaunch already unregistered AMs on RM restart (was: RM state store not cleaned if job succeeds but RM shutdown and restart-dispatcher stopped before it can process REMOVE_APP event) Race condition causing RM to potentially relaunch already unregistered AMs on RM restart Key: YARN-540 URL: https://issues.apache.org/jira/browse/YARN-540 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Jian He Priority: Blocker Attachments: YARN-540.patch, YARN-540.patch When job succeeds and successfully call finishApplicationMaster, RM shutdown and restart-dispatcher is stopped before it can process REMOVE_APP event. The next time RM comes back, it will reload the existing state files even though the job is succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1024: - Attachment: CPUasaYARNresource.pdf Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: CPUasaYARNresource.pdf We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748062#comment-13748062 ] Sandy Ryza commented on YARN-1024: -- I wrote up a more detailed proposal and attached a PDF of it. Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: CPUasaYARNresource.pdf We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-1074:
Attachment: YARN-1074.5.patch
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748067#comment-13748067
]
Xuan Gong commented on YARN-1074:
-
bq.TestApplicationACLs changes seem unnecessary. Shouldn't the default
behaviour be enough for this?
Yes, we do not need to make any changes. Revert the changes from
TestApplicationACLs and TestClientRMServices.
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748068#comment-13748068
]
Xuan Gong commented on YARN-1074:
-
Use EnumSetYarnApplicationState in the new patch.
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-556) RM Restart phase 2 - Work preserving restart
[ https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-556: Attachment: Work Preserving RM Restart.pdf Attaching a proposal with details. I may have missed writing something even though I thought of it or may have missed something altogether. Will incorporate feedback as it comes. Will soon start creating sub-tasks that make sense in a chronological ordering of work. Making incremental progress while keeping the RM stable is the desired course of action (like YARN-128). RM Restart phase 2 - Work preserving restart Key: YARN-556 URL: https://issues.apache.org/jira/browse/YARN-556 Project: Hadoop YARN Issue Type: New Feature Components: resourcemanager Reporter: Bikas Saha Assignee: Bikas Saha Attachments: Work Preserving RM Restart.pdf YARN-128 covered storing the state needed for the RM to recover critical information. This umbrella jira will track changes needed to recover the running state of the cluster so that work can be preserved across RM restarts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-707: - Attachment: YARN-707-20130822.txt Here's a quick patch to add user info to the ClientToAMToken. Also added a test to verify denial to tampering of user info. Have to do sanity single node tests. Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Vinod Kumar Vavilapalli Fix For: 2.1.0-beta Attachments: YARN-707-20130822.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-1074:
Attachment: YARN-1074.6.patch
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748241#comment-13748241
]
Xuan Gong commented on YARN-1074:
-
New patch addresses all the comments
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
