[jira] [Commented] (YARN-1772) Fair Scheduler documentation should indicate that admin ACLs also give submit permissions
[ https://issues.apache.org/jira/browse/YARN-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524523#comment-14524523 ] Naren Koneru commented on YARN-1772: Hi Jian, I won't be able to.. Feel free to take it.. > Fair Scheduler documentation should indicate that admin ACLs also give submit > permissions > - > > Key: YARN-1772 > URL: https://issues.apache.org/jira/browse/YARN-1772 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Naren Koneru > > I can submit to a Fair Scheduler queue if I'm in the submit ACL OR if I'm in > the administer ACL. The Fair Scheduler docs seem to leave out the second > part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13938366#comment-13938366 ] Naren Koneru commented on YARN-1577: Hi Jian, Sorry I am stuck with a few fires here and would be busy the next couple of days. Would you be able to take this jira if you need it prior to that?...Pls let me know and sorry about that !. I owe you one :-) > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13938170#comment-13938170 ] Naren Koneru commented on YARN-1577: Sorry guys, been busy with some internal release stuff here. Will try to get it later today and if I cannot, will find someone to fix it...Thanks !... > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933541#comment-13933541 ] Naren Koneru commented on YARN-1577: Hi Zhijie, Nice, thanks for letting me know. I will use that in llama and also submit a patch for yarn unmanagedamlauncher later today. regards Naren > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915243#comment-13915243 ] Naren Koneru commented on YARN-1577: I see that a patch has been submitted today for YARN-1389. I will look into this once that issue is resolved since we are gonna make use of the APIs there (getApplicationAttemptReport) to fi the UnManagedAMLauncher, etc > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914853#comment-13914853 ] Naren Koneru commented on YARN-1577: Vinod, Jian.. thanks for the inputs.. I will look into this further and submit a patch > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913770#comment-13913770 ] Naren Koneru commented on YARN-1577: Agreed that when people use UnmanagedAMLauncher, the change can be hidden. However, in some cases, we dont use this class since we dont necessarily spawn a process for AM in the unmanaged client always. I am ok to break compatibility for this use case and we can fix the UnmanagedAMLauncher to wait till the attempt is launched. As for the hack you mentioned, though that solution seems to be the least intrusive, I am a little skeptical if this would have any side effects since we would be lying in the ApplicationClientProtocol. Otherwise, we can get the changes from YARN-1389 and change the UnmanagedAMLauncher and the llama clients accordingly. thoughts??..I am ok with either approach. > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913633#comment-13913633 ] Naren Koneru commented on YARN-1577: After digging through the details, here's the summary as I understand (sorry for the repetition if any). - Today, the unmanaged client (llama) is sending a request to launch the AM, then waiting for the App state to be ACCEPTED and then its registering the AM using AMRMClientAsync.registerApplicationMaster. - This register call expects the AM RM token to be set, which is part of the application report. The Client gets this token by calling ApplicationClientProtocol.getApplicationReport after the APP is accepted. With the change in YARN-1493, this is broken since the AppAttempt is launched after the application is accepted and hence the token is not set. So the client can run into race conditions depending on when its getting the application report. The temporary hack we made in the client is to retry for a fixed number of times. One way to solve this could be: - Change the ApplicationReport (returned by ApplicationClientProtocol.getApplicationReport) to add an attempt state, so the client can rely on the Attempt state to be launched before proceeding with the UAM registration. - However, this would not be backwards compatible since it involves changes to the unmanaged clients. Since I do not see any documentation for the unmanaged clients, is this acceptable? Is this proposal ok?. If not, any other suggestions?. If this proposal is ok, then I can submit a patch. Pls comment. > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru reassigned YARN-1577: -- Assignee: Naren Koneru (was: Jian He) > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911876#comment-13911876 ] Naren Koneru commented on YARN-1577: Hi Jian, are you working on this issue? If not, I would like to take a look. Can you please comment. > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1745) Add yarn.dispatcher.exit-on-error to yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru reassigned YARN-1745: -- Assignee: Naren Koneru (was: Sandy Ryza) > Add yarn.dispatcher.exit-on-error to yarn-default.xml > - > > Key: YARN-1745 > URL: https://issues.apache.org/jira/browse/YARN-1745 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Naren Koneru > > This property is handled in code really weirdly. There's a default value of > false supplied in Dispatcher.java, but everywhere that uses it in the code > explicitly sets it to true. > We should change the default in Dispatcher.java to true and add it with this > same value to yarn-default.xml. > We'll also need a MapReduce JIRA for making the relevant changes in the > JobHistoryServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1736) In Fair Scheduler, AppSchedulable.assignContainer Priority argument is redundant with ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907408#comment-13907408 ] Naren Koneru commented on YARN-1736: The change was made to an internal private function to remove a parameter and use it from elsewhere. So no new tests are required. I ran the TestFairScheduler test case manually. > In Fair Scheduler, AppSchedulable.assignContainer Priority argument is > redundant with ResourceRequest > - > > Key: YARN-1736 > URL: https://issues.apache.org/jira/browse/YARN-1736 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza >Priority: Minor > Attachments: YARN-1736-1.patch > > > The ResourceRequest includes a Priority, so no need to pass in a Priority > alongside it -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1736) In Fair Scheduler, AppSchedulable.assignContainer Priority argument is redundant with ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1736: --- Attachment: YARN-1736-1.patch > In Fair Scheduler, AppSchedulable.assignContainer Priority argument is > redundant with ResourceRequest > - > > Key: YARN-1736 > URL: https://issues.apache.org/jira/browse/YARN-1736 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza >Priority: Minor > Attachments: YARN-1736-1.patch > > > The ResourceRequest includes a Priority, so no need to pass in a Priority > alongside it -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1736) In Fair Scheduler, AppSchedulable.assignContainer Priority argument is redundant with ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru reassigned YARN-1736: -- Assignee: Naren Koneru > In Fair Scheduler, AppSchedulable.assignContainer Priority argument is > redundant with ResourceRequest > - > > Key: YARN-1736 > URL: https://issues.apache.org/jira/browse/YARN-1736 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Naren Koneru >Priority: Minor > > The ResourceRequest includes a Priority, so no need to pass in a Priority > alongside it -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1171) Add default queue properties to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906100#comment-13906100 ] Naren Koneru commented on YARN-1171: Changed the issue to reflect the current state of code vs documentation and fixed the documentation. > Add default queue properties to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Karthik Kambatla > Attachments: YARN-1171-1.patch > > > The Fair Scheduler doc is missing the following properties. > - defaultMinSharePreemptionTimeout > - queueMaxAppsDefault -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1171) Add default queue properties to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1171: --- Attachment: YARN-1171-1.patch > Add default queue properties to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Karthik Kambatla > Attachments: YARN-1171-1.patch > > > The Fair Scheduler doc is missing the following properties. > - defaultMinSharePreemptionTimeout > - queueMaxAppsDefault -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1171) Add default queue properties to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1171: --- Description: The Fair Scheduler doc is missing the following properties. - defaultMinSharePreemptionTimeout - queueMaxAppsDefault was: The Fair Scheduler doc is missing the following properties. - defaultMinSharePreemptionTimeout - queueMaxAppsDefault - queuePlacementPolicy > Add default queue properties to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Naren Koneru > > The Fair Scheduler doc is missing the following properties. > - defaultMinSharePreemptionTimeout > - queueMaxAppsDefault -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1171) Add default queue properties to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1171: --- Description: The Fair Scheduler doc is missing the following properties. - defaultMinSharePreemptionTimeout - queueMaxAppsDefault - queuePlacementPolicy was:The Fair Scheduler doc is missing the defaultQueueSchedulingPolicy property. I suspect there are a few other ones too that provide defaults for all queues. > Add default queue properties to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Naren Koneru > > The Fair Scheduler doc is missing the following properties. > - defaultMinSharePreemptionTimeout > - queueMaxAppsDefault > - queuePlacementPolicy -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1171) Add default queue properties to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1171: --- Summary: Add default queue properties to Fair Scheduler documentation (was: Add defaultQueueSchedulingPolicy to Fair Scheduler documentation ) > Add default queue properties to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Naren Koneru > > The Fair Scheduler doc is missing the defaultQueueSchedulingPolicy property. > I suspect there are a few other ones too that provide defaults for all queues. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1171) Add defaultQueueSchedulingPolicy to Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru reassigned YARN-1171: -- Assignee: Naren Koneru (was: Karthik Kambatla) > Add defaultQueueSchedulingPolicy to Fair Scheduler documentation > - > > Key: YARN-1171 > URL: https://issues.apache.org/jira/browse/YARN-1171 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, scheduler >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Naren Koneru > > The Fair Scheduler doc is missing the defaultQueueSchedulingPolicy property. > I suspect there are a few other ones too that provide defaults for all queues. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1672) YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds
[ https://issues.apache.org/jira/browse/YARN-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naren Koneru updated YARN-1672: --- Attachment: YARN-1672-1.patch Add the default in java code and change the places where we used the constant to use this parameter. > YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds > -- > > Key: YARN-1672 > URL: https://issues.apache.org/jira/browse/YARN-1672 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Naren Koneru >Priority: Trivial > Attachments: YARN-1672-1.patch > > > YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds -- This message was sent by Atlassian JIRA (v6.1.5#6160)
