[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748241#comment-13748241
]
Xuan Gong commented on YARN-1074:
-
New patch addresses all the comments
> Clean up YARN CLI app list to show only running apps.
> -
>
> Key: YARN-1074
> URL: https://issues.apache.org/jira/browse/YARN-1074
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
> YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch
>
>
> Once a user brings up YARN daemon, runs jobs, jobs will stay in output
> returned by $ yarn application -list even after jobs complete already. We
> want YARN command line to clean up this list. Specifically, we want to remove
> applications with FINISHED state(not Final-State) or KILLED state from the
> result.
> {code}
> [user1@host1 ~]$ yarn application -list
> Total Applications:150
> Application-IdApplication-Name
> Application-Type User Queue State
> Final-State ProgressTracking-URL
> application_1374638600275_0109 Sleep job
> MAPREDUCEuser1 default KILLED
> KILLED 100%host1:54059
> application_1374638600275_0121 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
> application_1374638600275_0020 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
> application_1374638600275_0038 Sleep job
> MAPREDUCEuser1 default
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-1074:
Attachment: YARN-1074.6.patch
> Clean up YARN CLI app list to show only running apps.
> -
>
> Key: YARN-1074
> URL: https://issues.apache.org/jira/browse/YARN-1074
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
> YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch
>
>
> Once a user brings up YARN daemon, runs jobs, jobs will stay in output
> returned by $ yarn application -list even after jobs complete already. We
> want YARN command line to clean up this list. Specifically, we want to remove
> applications with FINISHED state(not Final-State) or KILLED state from the
> result.
> {code}
> [user1@host1 ~]$ yarn application -list
> Total Applications:150
> Application-IdApplication-Name
> Application-Type User Queue State
> Final-State ProgressTracking-URL
> application_1374638600275_0109 Sleep job
> MAPREDUCEuser1 default KILLED
> KILLED 100%host1:54059
> application_1374638600275_0121 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
> application_1374638600275_0020 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
> application_1374638600275_0038 Sleep job
> MAPREDUCEuser1 default
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-707: - Attachment: YARN-707-20130822.txt Here's a quick patch to add user info to the ClientToAMToken. Also added a test to verify denial to tampering of user info. Have to do sanity single node tests. > Add user info in the YARN ClientToken > - > > Key: YARN-707 > URL: https://issues.apache.org/jira/browse/YARN-707 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Vinod Kumar Vavilapalli > Fix For: 2.1.0-beta > > Attachments: YARN-707-20130822.txt > > > If user info is present in the client token then it can be used to do limited > authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-556) RM Restart phase 2 - Work preserving restart
[ https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-556: Attachment: Work Preserving RM Restart.pdf Attaching a proposal with details. I may have missed writing something even though I thought of it or may have missed something altogether. Will incorporate feedback as it comes. Will soon start creating sub-tasks that make sense in a chronological ordering of work. Making incremental progress while keeping the RM stable is the desired course of action (like YARN-128). > RM Restart phase 2 - Work preserving restart > > > Key: YARN-556 > URL: https://issues.apache.org/jira/browse/YARN-556 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: Work Preserving RM Restart.pdf > > > YARN-128 covered storing the state needed for the RM to recover critical > information. This umbrella jira will track changes needed to recover the > running state of the cluster so that work can be preserved across RM restarts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748068#comment-13748068
]
Xuan Gong commented on YARN-1074:
-
Use EnumSet in the new patch.
> Clean up YARN CLI app list to show only running apps.
> -
>
> Key: YARN-1074
> URL: https://issues.apache.org/jira/browse/YARN-1074
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
> YARN-1074.4.patch, YARN-1074.5.patch
>
>
> Once a user brings up YARN daemon, runs jobs, jobs will stay in output
> returned by $ yarn application -list even after jobs complete already. We
> want YARN command line to clean up this list. Specifically, we want to remove
> applications with FINISHED state(not Final-State) or KILLED state from the
> result.
> {code}
> [user1@host1 ~]$ yarn application -list
> Total Applications:150
> Application-IdApplication-Name
> Application-Type User Queue State
> Final-State ProgressTracking-URL
> application_1374638600275_0109 Sleep job
> MAPREDUCEuser1 default KILLED
> KILLED 100%host1:54059
> application_1374638600275_0121 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
> application_1374638600275_0020 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
> application_1374638600275_0038 Sleep job
> MAPREDUCEuser1 default
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748067#comment-13748067
]
Xuan Gong commented on YARN-1074:
-
bq.TestApplicationACLs changes seem unnecessary. Shouldn't the default
behaviour be enough for this?
Yes, we do not need to make any changes. Revert the changes from
TestApplicationACLs and TestClientRMServices.
> Clean up YARN CLI app list to show only running apps.
> -
>
> Key: YARN-1074
> URL: https://issues.apache.org/jira/browse/YARN-1074
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
> YARN-1074.4.patch, YARN-1074.5.patch
>
>
> Once a user brings up YARN daemon, runs jobs, jobs will stay in output
> returned by $ yarn application -list even after jobs complete already. We
> want YARN command line to clean up this list. Specifically, we want to remove
> applications with FINISHED state(not Final-State) or KILLED state from the
> result.
> {code}
> [user1@host1 ~]$ yarn application -list
> Total Applications:150
> Application-IdApplication-Name
> Application-Type User Queue State
> Final-State ProgressTracking-URL
> application_1374638600275_0109 Sleep job
> MAPREDUCEuser1 default KILLED
> KILLED 100%host1:54059
> application_1374638600275_0121 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
> application_1374638600275_0020 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
> application_1374638600275_0038 Sleep job
> MAPREDUCEuser1 default
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-1074:
Attachment: YARN-1074.5.patch
> Clean up YARN CLI app list to show only running apps.
> -
>
> Key: YARN-1074
> URL: https://issues.apache.org/jira/browse/YARN-1074
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
> YARN-1074.4.patch, YARN-1074.5.patch
>
>
> Once a user brings up YARN daemon, runs jobs, jobs will stay in output
> returned by $ yarn application -list even after jobs complete already. We
> want YARN command line to clean up this list. Specifically, we want to remove
> applications with FINISHED state(not Final-State) or KILLED state from the
> result.
> {code}
> [user1@host1 ~]$ yarn application -list
> Total Applications:150
> Application-IdApplication-Name
> Application-Type User Queue State
> Final-State ProgressTracking-URL
> application_1374638600275_0109 Sleep job
> MAPREDUCEuser1 default KILLED
> KILLED 100%host1:54059
> application_1374638600275_0121 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
> application_1374638600275_0020 Sleep job
> MAPREDUCEuser1 defaultFINISHED
> SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
> application_1374638600275_0038 Sleep job
> MAPREDUCEuser1 default
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1024: - Attachment: CPUasaYARNresource.pdf > Define a virtual core unambigiously > --- > > Key: YARN-1024 > URL: https://issues.apache.org/jira/browse/YARN-1024 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: CPUasaYARNresource.pdf > > > We need to clearly define the meaning of a virtual core unambiguously so that > it's easy to migrate applications between clusters. > For e.g. here is Amazon EC2 definition of ECU: > http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it > Essentially we need to clearly define a YARN Virtual Core (YVC). > Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the > equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748062#comment-13748062 ] Sandy Ryza commented on YARN-1024: -- I wrote up a more detailed proposal and attached a PDF of it. > Define a virtual core unambigiously > --- > > Key: YARN-1024 > URL: https://issues.apache.org/jira/browse/YARN-1024 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: CPUasaYARNresource.pdf > > > We need to clearly define the meaning of a virtual core unambiguously so that > it's easy to migrate applications between clusters. > For e.g. here is Amazon EC2 definition of ECU: > http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it > Essentially we need to clearly define a YARN Virtual Core (YVC). > Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the > equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-540) Race condition causing RM to potentially relaunch already unregistered AMs on RM restart
[ https://issues.apache.org/jira/browse/YARN-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-540: - Summary: Race condition causing RM to potentially relaunch already unregistered AMs on RM restart (was: RM state store not cleaned if job succeeds but RM shutdown and restart-dispatcher stopped before it can process REMOVE_APP event) > Race condition causing RM to potentially relaunch already unregistered AMs on > RM restart > > > Key: YARN-540 > URL: https://issues.apache.org/jira/browse/YARN-540 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > Attachments: YARN-540.patch, YARN-540.patch > > > When job succeeds and successfully call finishApplicationMaster, RM shutdown > and restart-dispatcher is stopped before it can process REMOVE_APP event. The > next time RM comes back, it will reload the existing state files even though > the job is succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-540) RM state store not cleaned if job succeeds but RM shutdown and restart-dispatcher stopped before it can process REMOVE_APP event
[ https://issues.apache.org/jira/browse/YARN-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748024#comment-13748024 ] Vinod Kumar Vavilapalli commented on YARN-540: -- I think we should fix it the right way. And after things like changes in MAPREDUCE-5476, AM's spending time in FINISHING state is going to be more common. So, I am +1 to Bikas's proposal (2). But in the interesting of not making incompatible changes, let's do the following: - Let's change FinishApplicationMasterResponse to also container a response-completed field. If it is true, it means that RM has finished the finalization of AM, otherwise, AM is supposed to retry till it becomes true - Let RM do the state-store changes asynchronously. It'll still be a behavior change, but clients which don't follow the multi-step unregister will risk only getting restarted. > RM state store not cleaned if job succeeds but RM shutdown and > restart-dispatcher stopped before it can process REMOVE_APP event > > > Key: YARN-540 > URL: https://issues.apache.org/jira/browse/YARN-540 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > Attachments: YARN-540.patch, YARN-540.patch > > > When job succeeds and successfully call finishApplicationMaster, RM shutdown > and restart-dispatcher is stopped before it can process REMOVE_APP event. The > next time RM comes back, it will reload the existing state files even though > the job is succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1092) RM proxy URLs invalid when running without a YARN proxy service
[
https://issues.apache.org/jira/browse/YARN-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747896#comment-13747896
]
Steve Loughran commented on YARN-1092:
--
# My AM generates a URL to the hbase master, such as {{http://ubuntu:8080}}, a
URL registered with an AM that is not running a proxy service, and whose
yarn-site.xml does not declare one
# the link on the GUI is {{http://ubuntu:9081//ubuntu:8080}}
# Which leads to a 404 page
> RM proxy URLs invalid when running without a YARN proxy service
> ---
>
> Key: YARN-1092
> URL: https://issues.apache.org/jira/browse/YARN-1092
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
> Environment: single host cluster without any yarn proxy service
>Reporter: Steve Loughran
>Priority: Minor
>
> The URLs generated in the RM GUI aren't valid if you don't have a proxy
> server running
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1092) RM proxy URLs invalid when running without a YARN proxy service
Steve Loughran created YARN-1092: Summary: RM proxy URLs invalid when running without a YARN proxy service Key: YARN-1092 URL: https://issues.apache.org/jira/browse/YARN-1092 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Environment: single host cluster without any yarn proxy service Reporter: Steve Loughran Priority: Minor The URLs generated in the RM GUI aren't valid if you don't have a proxy server running -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747881#comment-13747881 ] Sandy Ryza commented on YARN-1089: -- Yeah, I'll write up a document and post it on YARN-1024. I'm hoping to keep the broader discussion there so we can use this (and perhaps additional JIRAs) for the actual implementation. > Add YARN compute units alongside virtual cores > -- > > Key: YARN-1089 > URL: https://issues.apache.org/jira/browse/YARN-1089 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > Based on discussion in YARN-1024, we will add YARN compute units as a > resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747855#comment-13747855 ] Hitesh Shah commented on YARN-1089: --- +1 to [~vinodkv]'s request. [~sandyr] Could you ensure that the proposal also clearly explains how an application developer is meant to use the compute units and/or virtual cores when defining an allocation request and how the allocation (based on these 2 params) will be enforced on a container. > Add YARN compute units alongside virtual cores > -- > > Key: YARN-1089 > URL: https://issues.apache.org/jira/browse/YARN-1089 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > Based on discussion in YARN-1024, we will add YARN compute units as a > resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747851#comment-13747851 ] Vinod Kumar Vavilapalli commented on YARN-1089: --- I haven't been following YARN-1024, can you please summarize the proposal here for discussion? Thanks. > Add YARN compute units alongside virtual cores > -- > > Key: YARN-1089 > URL: https://issues.apache.org/jira/browse/YARN-1089 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > Based on discussion in YARN-1024, we will add YARN compute units as a > resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747750#comment-13747750 ] Arun C Murthy commented on YARN-1089: - +1 for this enhancement. > Add YARN compute units alongside virtual cores > -- > > Key: YARN-1089 > URL: https://issues.apache.org/jira/browse/YARN-1089 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > Based on discussion in YARN-1024, we will add YARN compute units as a > resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-899) Get queue administration ACLs working
[ https://issues.apache.org/jira/browse/YARN-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-899: --- Attachment: YARN-899.6.patch create the patch based on the latest trunk > Get queue administration ACLs working > - > > Key: YARN-899 > URL: https://issues.apache.org/jira/browse/YARN-899 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Xuan Gong > Attachments: YARN-899.1.patch, YARN-899.2.patch, YARN-899.3.patch, > YARN-899.4.patch, YARN-899.5.patch, YARN-899.5.patch, YARN-899.6.patch > > > The Capacity Scheduler documents the > yarn.scheduler.capacity.root..acl_administer_queue config option > for controlling who can administer a queue, but it is not hooked up to > anything. The Fair Scheduler could make use of a similar option as well. > This is a feature-parity regression from MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1089) Add YARN compute units alongside virtual cores
[ https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1089: - Description: Based on discussion in YARN-1024, we will add YARN compute units as a resource for requesting and scheduling CPU processing power. > Add YARN compute units alongside virtual cores > -- > > Key: YARN-1089 > URL: https://issues.apache.org/jira/browse/YARN-1089 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > Based on discussion in YARN-1024, we will add YARN compute units as a > resource for requesting and scheduling CPU processing power. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747701#comment-13747701 ] Sandy Ryza commented on YARN-1024: -- Filed YARN-1089 for adding YCUs. > Define a virtual core unambigiously > --- > > Key: YARN-1024 > URL: https://issues.apache.org/jira/browse/YARN-1024 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > We need to clearly define the meaning of a virtual core unambiguously so that > it's easy to migrate applications between clusters. > For e.g. here is Amazon EC2 definition of ECU: > http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it > Essentially we need to clearly define a YARN Virtual Core (YVC). > Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the > equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1091) All containers localization fails in NM when any one of the configured nm local-dir disk becomes full
[
https://issues.apache.org/jira/browse/YARN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-1091.
--
Resolution: Duplicate
Duplicate of YARN-257.
> All containers localization fails in NM when any one of the configured nm
> local-dir disk becomes full
> -
>
> Key: YARN-1091
> URL: https://issues.apache.org/jira/browse/YARN-1091
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.5-alpha
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
>
> {code:xml}
> 2013-08-22 13:54:22,100 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
> create app directory
> /opt/nish/usercache/nish/appcache/application_1377151891396_0017
> java.io.IOException: mkdir of
> /opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
> 2013-08-22 13:54:22,102 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
> from
> /home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
> to
> /home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
> 2013-08-22 13:54:22,102 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
> to
> /home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
> =
> file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
> 2013-08-22 13:54:22,103 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.io.FileNotFoundException: File
> file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does
> not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
> 2013-08-22 13:54:22,104 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1377151891396_0017_01_000263 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747539#comment-13747539 ] Hudson commented on YARN-1082: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1526 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1526/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Secure RM with recovery enabled and rm state store on hdfs fails with gss > exception > --- > > Key: YARN-1082 > URL: https://issues.apache.org/jira/browse/YARN-1082 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Arpit Gupta >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747543#comment-13747543 ] Hudson commented on YARN-881: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1526 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1526/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java > Priority#compareTo method seems to be wrong. > > > Key: YARN-881 > URL: https://issues.apache.org/jira/browse/YARN-881 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.1-beta > > Attachments: YARN-881.1.patch, YARN-881.patch > > > if lower int value means higher priority, shouldn't we "return > other.getPriority() - this.getPriority() " -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747512#comment-13747512 ] Hudson commented on YARN-1082: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1499 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1499/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Secure RM with recovery enabled and rm state store on hdfs fails with gss > exception > --- > > Key: YARN-1082 > URL: https://issues.apache.org/jira/browse/YARN-1082 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Arpit Gupta >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747516#comment-13747516 ] Hudson commented on YARN-881: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1499 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1499/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java > Priority#compareTo method seems to be wrong. > > > Key: YARN-881 > URL: https://issues.apache.org/jira/browse/YARN-881 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.1-beta > > Attachments: YARN-881.1.patch, YARN-881.patch > > > if lower int value means higher priority, shouldn't we "return > other.getPriority() - this.getPriority() " -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.
[ https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747426#comment-13747426 ] Hudson commented on YARN-881: - SUCCESS: Integrated in Hadoop-Yarn-trunk #309 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/309/]) YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516331) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java > Priority#compareTo method seems to be wrong. > > > Key: YARN-881 > URL: https://issues.apache.org/jira/browse/YARN-881 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.1-beta > > Attachments: YARN-881.1.patch, YARN-881.patch > > > if lower int value means higher priority, shouldn't we "return > other.getPriority() - this.getPriority() " -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1082) Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
[ https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747422#comment-13747422 ] Hudson commented on YARN-1082: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #309 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/309/]) YARN-1082. Addendum patch. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516352) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java YARN-1082. Create base directories on HDFS after RM login to ensure RM recovery doesn't fail in secure mode. Contributed by Vinod K V. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1516337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Secure RM with recovery enabled and rm state store on hdfs fails with gss > exception > --- > > Key: YARN-1082 > URL: https://issues.apache.org/jira/browse/YARN-1082 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Arpit Gupta >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: YARN-1082-20130820.2.txt, YARN-1082-20130821.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1091) All containers localization fails in NM when any one of the configured nm local-dir disk becomes full
[
https://issues.apache.org/jira/browse/YARN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated YARN-1091:
Summary: All containers localization fails in NM when any one of the
configured nm local-dir disk becomes full (was: All containers localization
fails when any one of the configured nm local-dir disk becomes full)
> All containers localization fails in NM when any one of the configured nm
> local-dir disk becomes full
> -
>
> Key: YARN-1091
> URL: https://issues.apache.org/jira/browse/YARN-1091
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.5-alpha
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
>
> {code:xml}
> 2013-08-22 13:54:22,100 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
> create app directory
> /opt/nish/usercache/nish/appcache/application_1377151891396_0017
> java.io.IOException: mkdir of
> /opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
> 2013-08-22 13:54:22,102 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
> from
> /home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
> to
> /home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
> 2013-08-22 13:54:22,102 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
> to
> /home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
> =
> file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
> 2013-08-22 13:54:22,103 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.io.FileNotFoundException: File
> file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does
> not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
> 2013-08-22 13:54:22,104 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1377151891396_0017_01_000263 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1091) All containers localization fails when any one of the configured nm local-dir disk becomes full
Devaraj K created YARN-1091:
---
Summary: All containers localization fails when any one of the
configured nm local-dir disk becomes full
Key: YARN-1091
URL: https://issues.apache.org/jira/browse/YARN-1091
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
{code:xml}
2013-08-22 13:54:22,100 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
create app directory
/opt/nish/usercache/nish/appcache/application_1377151891396_0017
java.io.IOException: mkdir of
/opt/nish/usercache/nish/appcache/application_1377151891396_0017 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1125)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:426)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:90)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
from
/home/nish/new/JAN_4/nmlocal/nmPrivate/container_1377151891396_0017_01_000263.tokens
to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017/container_1377151891396_0017_01_000263.tokens
2013-08-22 13:54:22,102 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to
/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
=
file:/home/nish/new/JAN_4/nmlocal/usercache/nish/appcache/application_1377151891396_0017
2013-08-22 13:54:22,103 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.FileNotFoundException: File
file:/opt/nish/usercache/nish/appcache/application_1377151891396_0017 does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1112)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:391)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
2013-08-22 13:54:22,104 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1377151891396_0017_01_000263 transitioned from LOCALIZING
to LOCALIZATION_FAILED
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747319#comment-13747319 ] Mohammad Kamrul Islam commented on YARN-49: --- [~ojoshi] do you have WIP patch that i can use for new Giraph AM? It doesn't need to work though. > Improve distributed shell application to work on a secure cluster > - > > Key: YARN-49 > URL: https://issues.apache.org/jira/browse/YARN-49 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications/distributed-shell >Reporter: Hitesh Shah >Assignee: Omkar Vinit Joshi > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
