[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388210#comment-17388210
]
Daniel Templeton commented on YARN-10854:
-
The new description looks good. My only other ask would be that you also add a
sentence or two that addresses the bigger picture, e.g. the RM hangs onto
inactive nodes, but when you have a cluster that's being dynamically resized,
the RM needs to be able to let go of nodes that become inactive. Definitely
don't take that verbatim. :)
I'm no longer active enough in YARN to feel comfortable giving you a +1 in any
case, so you'll need to grab another committer for that. Add some motivational
text, and I'll give you my LGTM.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch, YARN-10854.002.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10854) Support marking inactive node as untracked without configured include path
[
https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384440#comment-17384440
]
Daniel Templeton commented on YARN-10854:
-
In principle the change looks fine to me. I'd like the docs in the XML file to
be a bit more explicit about what this property does. If you know about the
behavior of the include and exclude lists, the explanation is just barely
enough to know what the flag does. It would be better to be more verbose,
provide some context, make it easier to understand without reading source code.
> Support marking inactive node as untracked without configured include path
> --
>
> Key: YARN-10854
> URL: https://issues.apache.org/jira/browse/YARN-10854
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-10854.001.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a
> while(specified expiration time defined via
> {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60 seconds by
> default) and not exist in both include and exclude files can be marked as
> untracked nodes and can be removed from RM state (YARN-4311). It's very
> useful when auto-scaling is enabled in elastic cloud environment, which can
> avoid unlimited increase of inactive nodes (mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most
> of our cloud environments without configured white list of nodes, which can
> lead to easily control for the auto-scaling of nodes without further security
> requirements.
> So I propose to support marking inactive node as untracked without configured
> include path, to be compatible with the former versions, we can add a switch
> config for this.
> Any thoughts/suggestions/feedbacks are welcome!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
[
https://issues.apache.org/jira/browse/YARN-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069205#comment-17069205
]
Daniel Templeton commented on YARN-6886:
OK, I spent some time digging around in the code. It's been a while. :)
The issue with policies is ultimately resolved by
{{SchedulingPolicy.isChildPolicyAllowed()}} later in the process than
{{loadQueue()}}. I think it's fair to leave that one be.
{{maxResources}} could still use some help. If a queue is explicitly
configured with {{maxResources}} that are higher than those of its parent
queue, then we should be logging a warning message.
{{allowPreemptionFrom}} should also be checked, with a warning logged if a
queue tries to be preemptable when its parent isn't. A good place to log that
warning might be the end of {{FSQueue.updatePreemptionVariables()}}.
{{maxContainerAllocation}} and {{maxRunningApps}} should probably also have
warnings.
> AllocationFileLoaderService.loadQueue() should validate that setting do not
> conflict with parent
>
>
> Key: YARN-6886
> URL: https://issues.apache.org/jira/browse/YARN-6886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yousef Abu-Salah
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6886.001.diff
>
>
> Some settings, like policy, are limited by the queue's parent queue's
> configuration. We should check those settings when we load the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6551) Validate SLS input
[ https://issues.apache.org/jira/browse/YARN-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069193#comment-17069193 ] Daniel Templeton commented on YARN-6551: Try pulling the repo again and then reapplying your patch. > Validate SLS input > -- > > Key: YARN-6551 > URL: https://issues.apache.org/jira/browse/YARN-6551 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Major > Labels: newbie > Attachments: YARN-6551.001.diff > > > SLS takes three different input formats, SLS, RUMEN, and SYN. Some values > needs to be validated, e.g. node number cannot be negative. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5913) Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext
[ https://issues.apache.org/jira/browse/YARN-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069191#comment-17069191 ] Daniel Templeton commented on YARN-5913: Looks like you need to update your repo and try again. Reset the repo, pull it, and then reapply your patch. > Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext > -- > > Key: YARN-5913 > URL: https://issues.apache.org/jira/browse/YARN-5913 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Minor > Labels: newbie > Attachments: YARN-5913.001.diff > > > Usage of these two variables overlaps and causes confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
[
https://issues.apache.org/jira/browse/YARN-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069185#comment-17069185
]
Daniel Templeton edited comment on YARN-6886 at 3/28/20, 2:21 AM:
--
Nope, hasn't been fixed. It's just that the code where the issue lives has
been moved around. See {{AllocationFileQueueParser.lodQueue()}}.
was (Author: templedf):
Nope, hasn't been fixed. It's just that the code where the issue likes has
been moved around. See {{AllocationFileQueueParser.lodQueue()}}.
> AllocationFileLoaderService.loadQueue() should validate that setting do not
> conflict with parent
>
>
> Key: YARN-6886
> URL: https://issues.apache.org/jira/browse/YARN-6886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yousef Abu-Salah
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6886.001.diff
>
>
> Some settings, like policy, are limited by the queue's parent queue's
> configuration. We should check those settings when we load the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
[
https://issues.apache.org/jira/browse/YARN-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069185#comment-17069185
]
Daniel Templeton commented on YARN-6886:
Nope, hasn't been fixed. It's just that the code where the issue likes has
been moved around. See {{AllocationFileQueueParser.lodQueue()}}.
> AllocationFileLoaderService.loadQueue() should validate that setting do not
> conflict with parent
>
>
> Key: YARN-6886
> URL: https://issues.apache.org/jira/browse/YARN-6886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yousef Abu-Salah
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6886.001.diff
>
>
> Some settings, like policy, are limited by the queue's parent queue's
> configuration. We should check those settings when we load the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
[
https://issues.apache.org/jira/browse/YARN-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069178#comment-17069178
]
Daniel Templeton commented on YARN-6886:
[~ykabusalah], looks like you need to update your repo. The {{loadQueue()}}
method has been moved to {{AllocationFileQueueParser}} now.
> AllocationFileLoaderService.loadQueue() should validate that setting do not
> conflict with parent
>
>
> Key: YARN-6886
> URL: https://issues.apache.org/jira/browse/YARN-6886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yousef Abu-Salah
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6886.001.diff
>
>
> Some settings, like policy, are limited by the queue's parent queue's
> configuration. We should check those settings when we load the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5913) Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931077#comment-16931077
]
Daniel Templeton commented on YARN-5913:
The issue at hand is that in {{ApplicationSubmissionContext}} {{resource}}
({{getResource()}}/{{setResource()}}) has the same meaning as
{{AMContainerResourceRequests}}
({{getAMContainerResourceRequests()}}/{{setAMContainerResourceRequests()}}).
Having two different properties that do the same thing is confusing. The task
here is to pick one and replace all calls to the other with it. I would
recommend replacing all uses of {{resource}} with
{{AMContainerResourceRequests}}. Then mark {{getResource()}} and
{{setResource()}} as deprecated.
There's one more issue. In YARN-2493 the patch strips the {{@Stable}} tag off
{{getResource()}} and {{setResource()}}. That's not OK. The {{@Stable}}
should be restored.
Let me know if you need more details.
> Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext
> --
>
> Key: YARN-5913
> URL: https://issues.apache.org/jira/browse/YARN-5913
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Yufei Gu
>Assignee: Yousef Abu-Salah
>Priority: Minor
> Labels: newbie
>
> Usage of these two variables overlaps and causes confusion.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2497) Fair scheduler should support strict node labels
[ https://issues.apache.org/jira/browse/YARN-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926755#comment-16926755 ] Daniel Templeton commented on YARN-2497: You're welcome to play with the patch, but as noted, it's not complete. If you can live with the risks I outlined above, it might be OK to use. It almost certainly needs a rebase, though. > Fair scheduler should support strict node labels > > > Key: YARN-2497 > URL: https://issues.apache.org/jira/browse/YARN-2497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Wangda Tan >Assignee: Daniel Templeton >Priority: Major > Attachments: YARN-2497.001.patch, YARN-2497.002.patch, > YARN-2497.003.patch, YARN-2497.004.patch, YARN-2497.005.patch, > YARN-2497.006.patch, YARN-2497.007.patch, YARN-2497.008.patch, > YARN-2497.009.patch, YARN-2497.010.patch, YARN-2497.011.patch, > YARN-2497.branch-3.0.001.patch, YARN-2499.WIP01.patch > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5913) Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext
[ https://issues.apache.org/jira/browse/YARN-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-5913: -- Assignee: Yousef Abu-Salah > Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext > -- > > Key: YARN-5913 > URL: https://issues.apache.org/jira/browse/YARN-5913 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Minor > Labels: newbie > > Usage of these two variables overlaps and causes confusion. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6425) Move out FS state dump code out of method update()
[
https://issues.apache.org/jira/browse/YARN-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-6425:
--
Assignee: Yousef Abu-Salah
> Move out FS state dump code out of method update()
> --
>
> Key: YARN-6425
> URL: https://issues.apache.org/jira/browse/YARN-6425
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yousef Abu-Salah
>Priority: Major
> Labels: newbie++
>
> Better to move out FS state dump code out of update()
> {code}
> if (LOG.isDebugEnabled()) {
> if (--updatesToSkipForDebug < 0) {
> updatesToSkipForDebug = UPDATE_DEBUG_FREQUENCY;
> dumpSchedulerState();
> }
> }
> {code}
> And, after that we should distinct between update call and update thread
> duration like before YARN-6112.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894020#comment-16894020 ] Daniel Templeton commented on YARN-8480: I still hold that boolean resources are the way to go, but given that I'm no longer active in YARN, and that Yunicorn aspires to be the scheduler to rule them all, it's not worth the fight. Go ahead and close it. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8480.001.patch, YARN-8480.002.patch > > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6551) Validate SLS input
[ https://issues.apache.org/jira/browse/YARN-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-6551: -- Assignee: Yousef Abu-Salah > Validate SLS input > -- > > Key: YARN-6551 > URL: https://issues.apache.org/jira/browse/YARN-6551 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Major > Labels: newbie > > SLS takes three different input formats, SLS, RUMEN, and SYN. Some values > needs to be validated, e.g. node number cannot be negative. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
[ https://issues.apache.org/jira/browse/YARN-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-6886: -- Assignee: Yousef Abu-Salah > AllocationFileLoaderService.loadQueue() should validate that setting do not > conflict with parent > > > Key: YARN-6886 > URL: https://issues.apache.org/jira/browse/YARN-6886 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Yousef Abu-Salah >Priority: Minor > Labels: newbie > > Some settings, like policy, are limited by the queue's parent queue's > configuration. We should check those settings when we load the file. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6046) Documentation correction in YarnApplicationSecurity
[
https://issues.apache.org/jira/browse/YARN-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884376#comment-16884376
]
Daniel Templeton edited comment on YARN-6046 at 7/18/19 4:26 PM:
-
LGTM +1 . I'll commit it when I get a dev environment again (unless some
generous soul wants to do it for me).
was (Author: templedf):
LGTM +1 . I'll commit it when I get a dev environment again (unless some
generous sole wants to do it for me).
> Documentation correction in YarnApplicationSecurity
> ---
>
> Key: YARN-6046
> URL: https://issues.apache.org/jira/browse/YARN-6046
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Yousef Abu-Salah
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-6046.001.patch, yarn-6046.002.patch
>
>
> Few documentation correction required in
> hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
> {code}
> 1. Suring AM startup, log in to Kerberos.
> {code}
> {code}
> Don’t. Rely on the lifespan of the
> {code}
> {code}
> renewed automatically; the AM pushes out
> {code}
> {code}
> In an insecure cluster, the application will run as the identity of the
> account of the node manager, typically something such as yarn or mapred. By
> default, the application will access HDFS as that user, with a different home
> directory, and with a different user identified in audit logs and on file
> system owner attributes.
> {code}
> Need to reframe sentence.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6046) Documentation correction in YarnApplicationSecurity
[
https://issues.apache.org/jira/browse/YARN-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884376#comment-16884376
]
Daniel Templeton commented on YARN-6046:
LGTM +1 . I'll commit it when I get a dev environment again (unless some
generous sole wants to do it for me).
> Documentation correction in YarnApplicationSecurity
> ---
>
> Key: YARN-6046
> URL: https://issues.apache.org/jira/browse/YARN-6046
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Yousef Abu-Salah
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-6046.001.patch, yarn-6046.002.patch
>
>
> Few documentation correction required in
> hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
> {code}
> 1. Suring AM startup, log in to Kerberos.
> {code}
> {code}
> Don’t. Rely on the lifespan of the
> {code}
> {code}
> renewed automatically; the AM pushes out
> {code}
> {code}
> In an insecure cluster, the application will run as the identity of the
> account of the node manager, typically something such as yarn or mapred. By
> default, the application will access HDFS as that user, with a different home
> directory, and with a different user identified in audit logs and on file
> system owner attributes.
> {code}
> Need to reframe sentence.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6046) Documentation correction in YarnApplicationSecurity
[
https://issues.apache.org/jira/browse/YARN-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-6046:
--
Assignee: Yousef Abu-Salah (was: Prashant Jha)
> Documentation correction in YarnApplicationSecurity
> ---
>
> Key: YARN-6046
> URL: https://issues.apache.org/jira/browse/YARN-6046
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Yousef Abu-Salah
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-6046.001.patch
>
>
> Few documentation correction required in
> hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
> {code}
> 1. Suring AM startup, log in to Kerberos.
> {code}
> {code}
> Don’t. Rely on the lifespan of the
> {code}
> {code}
> renewed automatically; the AM pushes out
> {code}
> {code}
> In an insecure cluster, the application will run as the identity of the
> account of the node manager, typically something such as yarn or mapred. By
> default, the application will access HDFS as that user, with a different home
> directory, and with a different user identified in audit logs and on file
> system owner attributes.
> {code}
> Need to reframe sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797212#comment-16797212
]
Daniel Templeton commented on YARN-9398:
Oh, yeah. Forgot about the fact that there were also a bunch of warnings added
as well. If you can get a patch out that fixes the 7 warnings and the error
quickly, then let's do that. If not, I'd say let's get the error fixed now,
and fix the warnings in a lower-priority JIRA.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-9398.001.patch
>
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-9398:
---
Attachment: YARN-9398.001.patch
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-9398.001.patch
>
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797185#comment-16797185
]
Daniel Templeton commented on YARN-9398:
I went ahead and posted patch.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-9398.001.patch
>
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797185#comment-16797185
]
Daniel Templeton edited comment on YARN-9398 at 3/20/19 1:46 PM:
-
I went ahead and posted a patch.
was (Author: templedf):
I went ahead and posted patch.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-9398.001.patch
>
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-9398:
--
Assignee: Daniel Templeton (was: Peter Bacsko)
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-9398.001.patch
>
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797166#comment-16797166
]
Daniel Templeton commented on YARN-9398:
I see it now. You're right, [~eyang].
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796305#comment-16796305 ] Daniel Templeton commented on YARN-9358: I just did a super quick pass. There are a couple issues in basic formatting. The descriptions must all end in a period. Otherwise it's a javadoc error. Also, the param and returns tag descriptions should not start with a capital letter nor end with a period (I don't remember seeing extraneous periods, but just in case...) > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch, YARN-9358.004.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795995#comment-16795995
]
Daniel Templeton commented on YARN-9398:
I'm seeing a ton of errors in the javadoc when I do a dist build, and they're
not the ones you're seeing:
{noformat}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (module-javadocs) on
project hadoop-yarn-server-nodemanager: MavenReportException: Error while
generating Javadoc:
[ERROR] Exit code: 1 - Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java:392:
warning: no @param for credentials
[ERROR] protected void initializePipeline(ApplicationAttemptId
applicationAttemptId,
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:962:
warning: no @param for homeResponse
[ERROR] protected void reAttachUAMAndMergeRegisterResponse(
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:962:
warning: no @param for appId
[ERROR] protected void reAttachUAMAndMergeRegisterResponse(
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:1302:
warning: no @return
[ERROR] protected AllocateResponse generateBaseAllocationResponse() {
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java:30:
warning: no description for @throws
[ERROR]* @throws Exception
[ERROR] ^
...
[ERROR]
[ERROR] Command line was:
/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/../bin/javadoc
@options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in
'/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/apidocs'
dir.
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-nodemanager
{noformat}
I never build with javadocs, so maybe those issues have been there a while
without me noticing. How come I don't see the errors shown in this JIRA but
instead see different errors?
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/s
[jira] [Assigned] (YARN-9340) [Clean-up] Remove NULL check before instanceof in ResourceRequestSetKey
[ https://issues.apache.org/jira/browse/YARN-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-9340: -- Assignee: Shweta > [Clean-up] Remove NULL check before instanceof in ResourceRequestSetKey > --- > > Key: YARN-9340 > URL: https://issues.apache.org/jira/browse/YARN-9340 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Shweta >Assignee: Shweta >Priority: Minor > Attachments: YARN-9340.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780068#comment-16780068 ] Daniel Templeton commented on YARN-9322: And thanks to [~sunilg] for the review! > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch, > YARN-9322.004.patch, YARN-9322.005.patch, YARN-9322.006.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780059#comment-16780059 ] Daniel Templeton commented on YARN-9322: OK, then. +1 and let's commit. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch, > YARN-9322.004.patch, YARN-9322.005.patch, YARN-9322.006.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9323) FSLeafQueue#computeMaxAMResource does not override zero values for custom resources
[
https://issues.apache.org/jira/browse/YARN-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779861#comment-16779861
]
Daniel Templeton commented on YARN-9323:
Looking at {{FSLeafQueue.computeMaxAMResource()}}, I'm worried that we're
strewing the innards of the metrics class out all over the place. Instead,
could we maybe move most of {{computeMaxAMResource()}} into a "fill in" method
in {{QueueMetrics}} that takes the fair share as a parameter, clones it, and
fills in the values that are 0. For the sake of getting this JIRA in, let's do
that in a follow-up JIRA.
> FSLeafQueue#computeMaxAMResource does not override zero values for custom
> resources
> ---
>
> Key: YARN-9323
> URL: https://issues.apache.org/jira/browse/YARN-9323
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9323.001.patch, YARN-9323.002.patch,
> YARN-9323.003.patch, YARN-9323.004.patch, YARN-9323.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779827#comment-16779827
]
Daniel Templeton commented on YARN-9322:
OK, +1 from me. Please file a follow-up JIRA, though, to add JavaDoc for all
the new methods you added in {{FSQueueMetrics}}, e.g. {{getMaxAMShare()}},
{{getAMResourceUsage()}}, etc. You should probably make the JIRA to add all
the missing JavaDocs.
> Store metrics for custom resource types into FSQueueMetrics and query them in
> FairSchedulerQueueInfo
>
>
> Key: YARN-9322
> URL: https://issues.apache.org/jira/browse/YARN-9322
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2019-02-21 at 12.06.46.png,
> YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch,
> YARN-9322.004.patch, YARN-9322.005.patch, YARN-9322.006.patch
>
>
> YARN-8842 implemented storing and exposing of metrics of custom resources.
> FSQueueMetrics should have a similar implementation.
> All metrics stored in this class should have their custom resource
> counterpart.
> In a consequence of metrics were not stored for custom resource type,
> FairSchedulerQueueInfo haven't contained those values therefore the UI v1
> could not show them, obviously.
> See that gpu is missing from the value of "AM Max Resources" on the attached
> screenshot.
> Additionally, the callees of the following methods (in class
> FairSchedulerQueueInfo) should consider to query values for custom resource
> types too:
> getMaxAMShareMB
> getMaxAMShareVCores
> getAMResourceUsageMB
> getAMResourceUsageVCores
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779614#comment-16779614 ] Daniel Templeton commented on YARN-9322: YARN-9318 is in, so let's rebase this and deal with the conflicts. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch, > YARN-9322.004.patch, YARN-9322.005.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9318) Resources#multiplyAndRoundUp does not consider Resource Types
[ https://issues.apache.org/jira/browse/YARN-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779582#comment-16779582 ] Daniel Templeton commented on YARN-9318: The @param tag shouldn't have a colon after the name, but I can fix that when I commit. Otherwise, +1 pending Jenkins. > Resources#multiplyAndRoundUp does not consider Resource Types > - > > Key: YARN-9318 > URL: https://issues.apache.org/jira/browse/YARN-9318 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9318.001.patch, YARN-9318.002.patch, > YARN-9318.003.patch, YARN-9318.004.patch, YARN-9318.005.patch > > > org.apache.hadoop.yarn.util.resource.Resources#multiplyAndRoundUp only deals > with memory and vcores while computing the rounded value. It should also > consider custom Resource Types as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779522#comment-16779522 ] Daniel Templeton commented on YARN-9322: LGTM except that it will still conflict with YARN-9318. Let's get that one in first, and then we can resolve the conflict. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch, > YARN-9322.004.patch, YARN-9322.005.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9318) Resources#multiplyAndRoundUp does not consider Resource Types
[ https://issues.apache.org/jira/browse/YARN-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779516#comment-16779516 ] Daniel Templeton commented on YARN-9318: Nice. Love that you added JavaDocs. Could you finish out the JavaDocs with @param and @return? Then I'm pretty sure we're done. > Resources#multiplyAndRoundUp does not consider Resource Types > - > > Key: YARN-9318 > URL: https://issues.apache.org/jira/browse/YARN-9318 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9318.001.patch, YARN-9318.002.patch, > YARN-9318.003.patch, YARN-9318.004.patch > > > org.apache.hadoop.yarn.util.resource.Resources#multiplyAndRoundUp only deals > with memory and vcores while computing the rounded value. It should also > consider custom Resource Types as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9318) Resources#multiplyAndRoundUp does not consider Resource Types
[
https://issues.apache.org/jira/browse/YARN-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778721#comment-16778721
]
Daniel Templeton commented on YARN-9318:
That looks much cleaner. My last comment is that in the 002 patch, you had
{{multiplyAndRoundDown()}} as a wrapper of {{multiplyTo()}}. I'd make
{{multiplyTo()}} a wrapper for {{multiplyAndRoundDown()}} now. Otherwise, it
looks good.
> Resources#multiplyAndRoundUp does not consider Resource Types
> -
>
> Key: YARN-9318
> URL: https://issues.apache.org/jira/browse/YARN-9318
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9318.001.patch, YARN-9318.002.patch,
> YARN-9318.003.patch
>
>
> org.apache.hadoop.yarn.util.resource.Resources#multiplyAndRoundUp only deals
> with memory and vcores while computing the rounded value. It should also
> consider custom Resource Types as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778716#comment-16778716
]
Daniel Templeton commented on YARN-9322:
Thanks, [~snemeth]. Comments:
# I think this patch will conflict with YARN-9318 on {{multiplyAndRoundUp()}}
# I like the cleanup of the ordering of the getters in {{FSQueueMetrics}}, but
{{getSteadyFairShareVCores()}} is out of place.
# As always, I'd appreciate meaningful assert messages.
# For completeness, you might consider adding an assert for a resource type not
in {{res}} in the new tests in {{TestFSQueueMetrics}}.
Overall, looks good. I don't care for the idea of having this big blob of
non-metrics in the metrics object, but I don't see a better way to do it at the
moment.
> Store metrics for custom resource types into FSQueueMetrics and query them in
> FairSchedulerQueueInfo
>
>
> Key: YARN-9322
> URL: https://issues.apache.org/jira/browse/YARN-9322
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2019-02-21 at 12.06.46.png,
> YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch
>
>
> YARN-8842 implemented storing and exposing of metrics of custom resources.
> FSQueueMetrics should have a similar implementation.
> All metrics stored in this class should have their custom resource
> counterpart.
> In a consequence of metrics were not stored for custom resource type,
> FairSchedulerQueueInfo haven't contained those values therefore the UI v1
> could not show them, obviously.
> See that gpu is missing from the value of "AM Max Resources" on the attached
> screenshot.
> Additionally, the callees of the following methods (in class
> FairSchedulerQueueInfo) should consider to query values for custom resource
> types too:
> getMaxAMShareMB
> getMaxAMShareVCores
> getAMResourceUsageMB
> getAMResourceUsageVCores
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778716#comment-16778716
]
Daniel Templeton edited comment on YARN-9322 at 2/27/19 12:02 AM:
--
Thanks, [~snemeth]. Comments:
# This patch will conflict with YARN-9318 on {{multiplyAndRoundUp()}}
# I like the cleanup of the ordering of the getters in {{FSQueueMetrics}}, but
{{getSteadyFairShareVCores()}} is out of place.
# As always, I'd appreciate meaningful assert messages.
# For completeness, you might consider adding an assert for a resource type not
in {{res}} in the new tests in {{TestFSQueueMetrics}}.
Overall, looks good. I don't care for the idea of having this big blob of
non-metrics in the metrics object, but I don't see a better way to do it at the
moment.
was (Author: templedf):
Thanks, [~snemeth]. Comments:
# I think this patch will conflict with YARN-9318 on {{multiplyAndRoundUp()}}
# I like the cleanup of the ordering of the getters in {{FSQueueMetrics}}, but
{{getSteadyFairShareVCores()}} is out of place.
# As always, I'd appreciate meaningful assert messages.
# For completeness, you might consider adding an assert for a resource type not
in {{res}} in the new tests in {{TestFSQueueMetrics}}.
Overall, looks good. I don't care for the idea of having this big blob of
non-metrics in the metrics object, but I don't see a better way to do it at the
moment.
> Store metrics for custom resource types into FSQueueMetrics and query them in
> FairSchedulerQueueInfo
>
>
> Key: YARN-9322
> URL: https://issues.apache.org/jira/browse/YARN-9322
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2019-02-21 at 12.06.46.png,
> YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch
>
>
> YARN-8842 implemented storing and exposing of metrics of custom resources.
> FSQueueMetrics should have a similar implementation.
> All metrics stored in this class should have their custom resource
> counterpart.
> In a consequence of metrics were not stored for custom resource type,
> FairSchedulerQueueInfo haven't contained those values therefore the UI v1
> could not show them, obviously.
> See that gpu is missing from the value of "AM Max Resources" on the attached
> screenshot.
> Additionally, the callees of the following methods (in class
> FairSchedulerQueueInfo) should consider to query values for custom resource
> types too:
> getMaxAMShareMB
> getMaxAMShareVCores
> getAMResourceUsageMB
> getAMResourceUsageVCores
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9323) FSLeafQueue#computeMaxAMResource does not override zero values for custom resources
[
https://issues.apache.org/jira/browse/YARN-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778685#comment-16778685
]
Daniel Templeton edited comment on YARN-9323 at 2/26/19 11:31 PM:
--
Thanks, [~snemeth].
I don't understand why {{computeMaxAMResource()}} uses the fair share for CPU
and memory but custom resources come from the available resources in the root
queue. Shouldn't they all just come from the fair share, making all those
other changes unnecessary?
was (Author: templedf):
Thanks, [~snemeth]. Comments:
# The patch doesn't compile: {noformat}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-resourcemanager: Compilation failure: Compilation
failure:
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java:[226,3]
cannot find symbol
[ERROR] symbol: class FSQueueMetricsForCustomResources
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java:[227,12]
cannot find symbol
[ERROR] symbol: variable customResources
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java:[528,49]
incompatible types:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsForCustomResources.QueueMetricsCustomResource
cannot be converted to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsCustomResource{noformat}
# The {{QueueMetricsCustomResource}} is duplicated. It's there once as a
public class and once as an inner class of {{QueueMetricsForCustomResources}}.
# I don't understand why {{computeMaxAMResource()}} uses the fair share for CPU
and memory but custom resources come from the available resources in the root
queue. Shouldn't they all just come from the fair share, making all those
other changes unnecessary?
> FSLeafQueue#computeMaxAMResource does not override zero values for custom
> resources
> ---
>
> Key: YARN-9323
> URL: https://issues.apache.org/jira/browse/YARN-9323
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9323.001.patch, YARN-9323.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9323) FSLeafQueue#computeMaxAMResource does not override zero values for custom resources
[
https://issues.apache.org/jira/browse/YARN-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778685#comment-16778685
]
Daniel Templeton commented on YARN-9323:
Thanks, [~snemeth]. Comments:
# The patch doesn't compile: {noformat}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-resourcemanager: Compilation failure: Compilation
failure:
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java:[226,3]
cannot find symbol
[ERROR] symbol: class FSQueueMetricsForCustomResources
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java:[227,12]
cannot find symbol
[ERROR] symbol: variable customResources
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java:[528,49]
incompatible types:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsForCustomResources.QueueMetricsCustomResource
cannot be converted to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsCustomResource{noformat}
# The {{QueueMetricsCustomResource}} is duplicated. It's there once as a
public class and once as an inner class of {{QueueMetricsForCustomResources}}.
# I don't understand why {{computeMaxAMResource()}} uses the fair share for CPU
and memory but custom resources come from the available resources in the root
queue. Shouldn't they all just come from the fair share, making all those
other changes unnecessary?
> FSLeafQueue#computeMaxAMResource does not override zero values for custom
> resources
> ---
>
> Key: YARN-9323
> URL: https://issues.apache.org/jira/browse/YARN-9323
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9323.001.patch, YARN-9323.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9318) Resources#multiplyAndRoundUp does not consider Resource Types
[
https://issues.apache.org/jira/browse/YARN-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778600#comment-16778600
]
Daniel Templeton commented on YARN-9318:
Thanks for the patch, [~snemeth]. I have a few comments:
# Your {{applyFunctionOnValues(Resource lhs, Resource rhs, BiFunction valueFunction)}} method adds complication for no gain. You're
still duplicating the logic, so you're better off leaving
{{multiplyAndAddTo()}} as it was.
# I'm kinda on the fence on the use of the lambdas. You only ever pass in two
lambdas: {{(value) -> (long) (value * by)}} and {{(value) -> (long)
Math.ceil(value * by)}}. The alternative would be to pass in a boolean that
says whether to round up or down. It would be easier to read but less
flexible. I don't see the multiplication functions changing that much in the
future, so I would suggest the boolean route.
# You've changed the behavior of the {{multiply*()}} methods. In the original
code, if a resource type causes an error, the operation aborts. In the new
code it keeps going. I personally think both behaviors are wrong; I think the
exception should be ducked. Regardless, let's not change behavior arbitrarily.
# In {{TestResources}} you should fix the name of {{testMultipleRoundUp()}} to
be {{testMultiplyRoundUp()}}.
# In {{testMultiplyAndRoundUpCustomResources()}} it would be great to have some
assert messages. While you're at it, it would also be nice to have more useful
assert messages in {{testMultipleRoundUp()}}.
> Resources#multiplyAndRoundUp does not consider Resource Types
> -
>
> Key: YARN-9318
> URL: https://issues.apache.org/jira/browse/YARN-9318
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9318.001.patch, YARN-9318.002.patch
>
>
> org.apache.hadoop.yarn.util.resource.Resources#multiplyAndRoundUp only deals
> with memory and vcores while computing the rounded value. It should also
> consider custom Resource Types as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778589#comment-16778589 ] Daniel Templeton commented on YARN-9322: I just retriggered the tests. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9326) Fair Scheduler configuration defaults are not documented in case of min and maxResources
[
https://issues.apache.org/jira/browse/YARN-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777087#comment-16777087
]
Daniel Templeton commented on YARN-9326:
Thanks for catching that, [~adam.antal]. A couple of comments:
# "not specified resource" should be "unspecified resource"
# "will be set to maximum" should be "will be set to the maximum value for that
resource"
> Fair Scheduler configuration defaults are not documented in case of min and
> maxResources
>
>
> Key: YARN-9326
> URL: https://issues.apache.org/jira/browse/YARN-9326
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: docs, documentation, fairscheduler, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9326.001.patch
>
>
> The FairScheduler's configuration has the following defaults (from the code:
> javadoc):
> {noformat}
> In new style resources, any resource that is not specified will be set to
> missing or 0%, as appropriate. Also, in the new style resources, units are
> not allowed. Units are assumed from the resource manager's settings for the
> resources when the value isn't a percentage. The missing parameter is only
> used in the case of new style resources without percentages. With new style
> resources with percentages, any missing resources will be assumed to be 100%
> because percentages are only used with maximum resource limits.
> {noformat}
> This is not documented in the hadoop yarn site FairScheduler.html. It is
> quite intuitive, but still need to be documented though.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681743#comment-16681743
]
Daniel Templeton commented on YARN-8990:
Thanks for finding and fixing this one, [~wilfreds]! It could have been a
source of much unhappiness in 3.2.
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8990) FS: race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-8990:
---
Priority: Blocker (was: Major)
> FS: race condition in app submit and queue cleanup
> --
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Attachments: YARN-8990.001.patch
>
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8985) FSParentQueue: debug log missing when assigning container
[ https://issues.apache.org/jira/browse/YARN-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679315#comment-16679315 ] Daniel Templeton commented on YARN-8985: Doesn't look like Jenkins started yet, so I just kicked it off manually. > FSParentQueue: debug log missing when assigning container > - > > Key: YARN-8985 > URL: https://issues.apache.org/jira/browse/YARN-8985 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Minor > Attachments: YARN-8985.001.patch > > > Tracking assignments in the queue hierarchy is not possible at a DEBUG level > because the FSParentQueue does not log a node being offered to the queue. > This means that if a parent queue has no leaf queues then it will be > impossible to track the offering leaving a hole in the tracking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8985) FSParentQueue: debug log missing when assigning container
[ https://issues.apache.org/jira/browse/YARN-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679314#comment-16679314 ] Daniel Templeton commented on YARN-8985: LGTM +1 pending Jenkins. > FSParentQueue: debug log missing when assigning container > - > > Key: YARN-8985 > URL: https://issues.apache.org/jira/browse/YARN-8985 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Minor > Attachments: YARN-8985.001.patch > > > Tracking assignments in the queue hierarchy is not possible at a DEBUG level > because the FSParentQueue does not log a node being offered to the queue. > This means that if a parent queue has no leaf queues then it will be > impossible to track the offering leaving a hole in the tracking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8985) FSParentQueue: debug log missing when assigning container
[ https://issues.apache.org/jira/browse/YARN-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679352#comment-16679352 ] Daniel Templeton commented on YARN-8985: The findbugs warning is legit. No idea what build error or all of the test failures are about. > FSParentQueue: debug log missing when assigning container > - > > Key: YARN-8985 > URL: https://issues.apache.org/jira/browse/YARN-8985 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Minor > Attachments: YARN-8985.001.patch > > > Tracking assignments in the queue hierarchy is not possible at a DEBUG level > because the FSParentQueue does not log a node being offered to the queue. > This means that if a parent queue has no leaf queues then it will be > impossible to track the offering leaving a hole in the tracking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8681) Wrong error message in RM placement constraints check
Daniel Templeton created YARN-8681:
--
Summary: Wrong error message in RM placement constraints check
Key: YARN-8681
URL: https://issues.apache.org/jira/browse/YARN-8681
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 3.1.1, 3.2.0
Reporter: Daniel Templeton
In {{SingleConstraintAppPlacementAllocator.validateAndSetSchedulingRequest()}}
I see the following:
{code} if (singleConstraint.getMinCardinality() != 0
|| singleConstraint.getMaxCardinality() != 0) {
throwExceptionWithMetaInfo(
"Only support anti-affinity, which is: minCardinality=0, "
+ "maxCardinality=1");
}{code}
I think the error message should say {{"maxCardinality=0"}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8621) Add REST API tests for Resource Types fields for the apps/ endpoint
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584336#comment-16584336 ] Daniel Templeton commented on YARN-8621: I don't think we need to return the resource requests from the appattempt endpoint. That's just one of many things returned by the application endpoint that's not returned by appattempt. > Add REST API tests for Resource Types fields for the apps/ endpoint > -- > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8621.001.patch, YARN-8621.002.patch > > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7133) Clean up lock-try order in fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553202#comment-16553202
]
Daniel Templeton commented on YARN-7133:
For one, it's the format set as the standard by Sun. See
https://docs.oracle.com/javase/6/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html
> Clean up lock-try order in fair scheduler
> -
>
> Key: YARN-7133
> URL: https://issues.apache.org/jira/browse/YARN-7133
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: newbie
> Attachments: YARN-7133.001.patch
>
>
> There are many places that follow the pattern:{code}try {
> lock.lock();
> ...
> } finally {
> lock.unlock();
> }{code}
> There are a couple of reasons that's a bad idea. The correct pattern
> is:{code}lock.lock();
> try {
> ...
> } finally {
> lock.unlock();
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551228#comment-16551228 ] Daniel Templeton commented on YARN-8480: I read through both node attributes and rich placement constraints. Correct me if I'm wrong, but it looks to me like node attributes rely on the PCM from rich placement constraints for scheduling. It also looks to me like the PCM is an out-of-band scheduling process that skips the usual scheduling process. The amount of code that changed in YARN-7670 to support rich placement constraints in capacity scheduler was smallish, but the impact is fairly large. Yes, the patch for this JIRA does touch a lot of common files, but the changes are minor and non-invasive and will come with careful testing for performance impact. The patch already includes a huge amount of new unit tests (on top of what was added with resource types already) to make sure nothing is impacted as a result of this minor extension. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8480.001.patch, YARN-8480.002.patch > > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540885#comment-16540885 ] Daniel Templeton commented on YARN-8480: I don't disagree that in terms of general use cases there's overlap between boolean resources and node attributes. From the perspective of fair scheduler, however, there's a big difference. Boolean resources are a simple mechanism for a pure label resource that is a minor extension of existing capabilities. Node labels or node attributes will be a major integration effort. While much of this work touches the common code, what we're proposing here is not intended as a general solution. Capacity scheduler is where the work for node attributes is happening, and we don't propose to change that or muddy those waters. This JIRA is intended only to provide a pure label resource for fair scheduler that isn't as heavy. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8480.001.patch, YARN-8480.002.patch > > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535129#comment-16535129 ] Daniel Templeton commented on YARN-8480: A boolean resource wouldn't be consumed by its use. The node attributes work is definitely in the same vein, though much heavier weight. The other difference is that there's no dependency on node labels or placement constraints here, so the scheduler implementation cost is very low. There is some overlap with node attributes, but when we have three entirely distinct ways to specify resources (resources, node labels, and node attributes), that's not too surprising. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8480) Add boolean option for resources
Daniel Templeton created YARN-8480: -- Summary: Add boolean option for resources Key: YARN-8480 URL: https://issues.apache.org/jira/browse/YARN-8480 Project: Hadoop YARN Issue Type: Improvement Reporter: Daniel Templeton Assignee: Szilard Nemeth Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514617#comment-16514617 ] Daniel Templeton commented on YARN-7556: The findbugs and unit test issues are unrelated. I think the shaded client issues should also be unrelated since there are no client-side changes here. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch, YARN-7556.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514447#comment-16514447 ] Daniel Templeton commented on YARN-7556: Eeew. That's a lot of failures. It doesn't look like they're related to my patch, though. I retriggered the job to see if that helps. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch, YARN-7556.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511754#comment-16511754 ] Daniel Templeton commented on YARN-6995: I rebased to latest trunk. [~sunilg], I agree that the error message doesn't apply in those cases, but the exception will be caught in those cases so it won't matter. > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.005.patch, YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch, > YARN-6995.YARN-3926.004.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6995: --- Attachment: YARN-6995.005.patch > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.005.patch, YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch, > YARN-6995.YARN-3926.004.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6964:
---
Attachment: YARN-6964.012.patch
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-6964.001.patch, YARN-6964.002.patch,
> YARN-6964.003.patch, YARN-6964.004.patch, YARN-6964.005.patch,
> YARN-6964.006.patch, YARN-6964.007.patch, YARN-6964.008.patch,
> YARN-6964.009.patch, YARN-6964.010.patch, YARN-6964.011.patch,
> YARN-6964.012.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511728#comment-16511728
]
Daniel Templeton commented on YARN-6964:
I made an arbitrary choice to set demand to twice the usage. It's just a
number to use in tests.
Patch 12 is rebased to the latest trunk.
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-6964.001.patch, YARN-6964.002.patch,
> YARN-6964.003.patch, YARN-6964.004.patch, YARN-6964.005.patch,
> YARN-6964.006.patch, YARN-6964.007.patch, YARN-6964.008.patch,
> YARN-6964.009.patch, YARN-6964.010.patch, YARN-6964.011.patch,
> YARN-6964.012.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511692#comment-16511692 ] Daniel Templeton commented on YARN-7556: New patch addresses [~wilfreds] docs concerns and rebases to the latest trunk. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch, YARN-7556.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7556: --- Attachment: YARN-7556.007.patch > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch, YARN-7556.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7467) FSLeafQueue unnecessarily calls ComputeFairShares.computeShare() to calculate fair share for apps
[
https://issues.apache.org/jira/browse/YARN-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508639#comment-16508639
]
Daniel Templeton commented on YARN-7467:
[~zhuqi], a couple of comments on the patch from YARN-8406:
* The _for_ loop should have the open brace ('\{') on the same line.
* The _if_ should have a space before the paren.
* I didn't apply the patch to look at the code in context, but it's unclear to
me why you're only working with memory. What about CPU and any user-defined
resources?
* Taking the ceiling will result in over assigning shares. I don't remember at
the moment whether that was the case with the normal compute shares code.
* You should probably cast the running apps count to a double when you declare
it so you can skip the casts in the line that does the math.
> FSLeafQueue unnecessarily calls ComputeFairShares.computeShare() to calculate
> fair share for apps
> -
>
> Key: YARN-7467
> URL: https://issues.apache.org/jira/browse/YARN-7467
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> All apps have the same weight, the same max share (unbounded), and the same
> min share (none). There's no reason to call {{computeShares()}} at all.
> Just divide the resources by the number of apps.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7467) FSLeafQueue unnecessarily calls ComputeFairShares.computeShare() to calculate fair share for apps
[
https://issues.apache.org/jira/browse/YARN-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508628#comment-16508628
]
Daniel Templeton commented on YARN-7467:
Go for it.
> FSLeafQueue unnecessarily calls ComputeFairShares.computeShare() to calculate
> fair share for apps
> -
>
> Key: YARN-7467
> URL: https://issues.apache.org/jira/browse/YARN-7467
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> All apps have the same weight, the same max share (unbounded), and the same
> min share (none). There's no reason to call {{computeShares()}} at all.
> Just divide the resources by the number of apps.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6132) SliderClient bondToCluster should call findInstance with live states filter
[ https://issues.apache.org/jira/browse/YARN-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508193#comment-16508193 ] Daniel Templeton commented on YARN-6132: Can be add a fixed in version for this issue? > SliderClient bondToCluster should call findInstance with live states filter > --- > > Key: YARN-6132 > URL: https://issues.apache.org/jira/browse/YARN-6132 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: YARN-6132-yarn-native-services.001.patch > > > _SliderClient#bondToCluster_ is called for apps which are in running state > only. Hence the call to findInstance should be filtered down by specifying > the set of live app states, rather than searching for apps in all states. > This makes the call more efficient and reduces "large response" data returned > by RM (which potentially causes more RM gc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4353) Provide short circuit user group mapping for NM/AM
[
https://issues.apache.org/jira/browse/YARN-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved YARN-4353.
Resolution: Won't Fix
Hadoop Flags: Reviewed
I'm fine with closing this out. I added {{NullGroupsMapping}} in order to
resolve this JIRA, but I never felt confident enough to pull the trigger.
> Provide short circuit user group mapping for NM/AM
> --
>
> Key: YARN-4353
> URL: https://issues.apache.org/jira/browse/YARN-4353
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: YARN-4353.prelim.patch
>
>
> When the NM launches an AM, the {{ContainerLocalizer}} gets the current user
> from {{UserGroupInformation}}, which triggers user group mapping, even though
> the user groups are never accessed. If secure LDAP is configured for group
> mapping, then there are some additional complications created by the
> unnecessary group resolution. Additionally, it adds unnecessary latency to
> the container launch time.
> To address the issue, before getting the current user, the
> {{ContainerLocalizer}} should configure {{UserGroupInformation}} with a null
> group mapping service that quickly and quietly returns an empty group list
> for all users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8200) Backport resource types/GPU features to branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450136#comment-16450136 ] Daniel Templeton edited comment on YARN-8200 at 4/24/18 4:05 PM: - The challenge in backporting resource types into 3.0 was mostly just in splitting resource types from resource profiles. Otherwise it wasn't bad. But that was pull from 3.x into 3.0. Going back into 2.x will be much trickier. The code that resource types touches is code that you want to handle very, very carefully because it's at the core of what the resource manager does. I don't think it's a good idea to pull resource types back into 2.x. Resource types represent a major change to the way the resource manager functions. It's not something that appropriate for a minor release. In fact, I would argue that resource types is one of the scarier changes in 3.0, so if you're willing to take on that risk in 2.x, you're probably better served just moving to 3.0. was (Author: templedf): The challenge in backporting resource types into 3.0 was mostly just in splitting resource types from resource profiles. Otherwise it wasn't bad. But that was pull from 3.x into 3.0. Going back into 2.x will be much trickier. The code that resource types touches is code that you want to handle very, very carefully because it's at the core of what the resource manager does. I don't think it's a good idea to pull resource types back into 2.x Resource types represent a major change to the way the resource manager functions. It's not something that appropriate for a minor release. In fact, I would argue that resource types is one of the scarier changes in 3.0, so if you're willing to take on that risk in 2.x, you're probably better served just moving to 3.0. > Backport resource types/GPU features to branch-2 > > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8200) Backport resource types/GPU features to branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450136#comment-16450136 ] Daniel Templeton edited comment on YARN-8200 at 4/24/18 4:05 PM: - The challenge in backporting resource types into 3.0 was mostly just in splitting resource types from resource profiles. Otherwise it wasn't bad. But that was pull from 3.x into 3.0. Going back into 2.x will be much trickier. The code that resource types touches is code that you want to handle very, very carefully because it's at the core of what the resource manager does. I don't think it's a good idea to pull resource types back into 2.x. Resource types represent a major change to the way the resource manager functions. It's not a change that's appropriate for a minor release. In fact, I would argue that resource types is one of the scarier changes in 3.0, so if you're willing to take on that risk in 2.x, you're probably better served just moving to 3.0. was (Author: templedf): The challenge in backporting resource types into 3.0 was mostly just in splitting resource types from resource profiles. Otherwise it wasn't bad. But that was pull from 3.x into 3.0. Going back into 2.x will be much trickier. The code that resource types touches is code that you want to handle very, very carefully because it's at the core of what the resource manager does. I don't think it's a good idea to pull resource types back into 2.x. Resource types represent a major change to the way the resource manager functions. It's not something that appropriate for a minor release. In fact, I would argue that resource types is one of the scarier changes in 3.0, so if you're willing to take on that risk in 2.x, you're probably better served just moving to 3.0. > Backport resource types/GPU features to branch-2 > > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450136#comment-16450136 ] Daniel Templeton commented on YARN-8200: The challenge in backporting resource types into 3.0 was mostly just in splitting resource types from resource profiles. Otherwise it wasn't bad. But that was pull from 3.x into 3.0. Going back into 2.x will be much trickier. The code that resource types touches is code that you want to handle very, very carefully because it's at the core of what the resource manager does. I don't think it's a good idea to pull resource types back into 2.x Resource types represent a major change to the way the resource manager functions. It's not something that appropriate for a minor release. In fact, I would argue that resource types is one of the scarier changes in 3.0, so if you're willing to take on that risk in 2.x, you're probably better served just moving to 3.0. > Backport resource types/GPU features to branch-2 > > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[ https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361805#comment-16361805 ] Daniel Templeton commented on YARN-7292: Hang on. If I understand that correctly, you're saying that in order to not have the profile overridden, the app developer has to create their own AMRMClient implementation? Doesn't that mean that profiles will always be overridden in all practical cases, e.g. MR, Spark, Hive, etc.? > Revisit Resource Profile Behavior > - > > Key: YARN-7292 > URL: https://issues.apache.org/jira/browse/YARN-7292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7292.002.patch, YARN-7292.003.patch, > YARN-7292.004.patch, YARN-7292.005.patch, YARN-7292.006.patch, > YARN-7292.wip.001.patch > > > Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a > couple of resource profile related behaviors might need to be updated: > 1) Configure resource profile in server side or client side: > Currently resource profile can be only configured centrally: > - Advantages: > A given resource profile has a the same meaning in the cluster. It won’t > change when we run different apps in different configurations. A job can run > under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit > is YARN scheduler can potentially do better bin packing. > - Disadvantages: > Hard for applications to add their own resource profiles. > 2) Do we really need mandatory resource profiles such as > minimum/maximum/default? > 3) Should we send resource profile name inside ResourceRequest, or should > client/AM translate it to resource and set it to the existing resource > fields? > 4) Related to above, should we allow resource overrides or client/AM should > send final resource to RM? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[ https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361217#comment-16361217 ] Daniel Templeton commented on YARN-7292: Sorry, still running way behind. Looking at the patch, it looks like you're ripping profiles out as an independent entity, presumably replacing them with resources. I think that largely makes sense. What's the plan with the overrides? We had issues with the current approach because MR *always* specifies the memory and CPU, so if we assume that a specified value overrides the profile, the profile will always be overridden. > Revisit Resource Profile Behavior > - > > Key: YARN-7292 > URL: https://issues.apache.org/jira/browse/YARN-7292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7292.002.patch, YARN-7292.003.patch, > YARN-7292.004.patch, YARN-7292.005.patch, YARN-7292.006.patch, > YARN-7292.wip.001.patch > > > Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a > couple of resource profile related behaviors might need to be updated: > 1) Configure resource profile in server side or client side: > Currently resource profile can be only configured centrally: > - Advantages: > A given resource profile has a the same meaning in the cluster. It won’t > change when we run different apps in different configurations. A job can run > under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit > is YARN scheduler can potentially do better bin packing. > - Disadvantages: > Hard for applications to add their own resource profiles. > 2) Do we really need mandatory resource profiles such as > minimum/maximum/default? > 3) Should we send resource profile name inside ResourceRequest, or should > client/AM translate it to resource and set it to the existing resource > fields? > 4) Related to above, should we allow resource overrides or client/AM should > send final resource to RM? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7819) Allow PlacementProcessor to be used with the FairScheduler
[
https://issues.apache.org/jira/browse/YARN-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347834#comment-16347834
]
Daniel Templeton commented on YARN-7819:
Comments:
* The write lock should be locked outside the {{try}} in {{allocateOnNode()}}
* Can we please move {{setQueueName()}} up to {{RMContainer}}? Every single
use of it involves a cast from {{RMContainer}} to {{RMContainerImpl}}, and
there are no other subclasses of {{RMContainer}}.
* Will calling the assignment node-local in the metrics update confuse the
metrics? What if it's not actually node local?
* The log messages on FSLeafQueue:360,364 is pretty cryptic. Either provide a
clear explanation of what happened and what should be done about it or consider
making then info or debug messages (or both!)
* The log message on FairScheduler:1879 definitely shouldn't be a warn. See
story as my previous point.
* Seems like there should be a cleaner way to do this:{code}Resource
resource =
schedulingRequest.getResourceSizing().getResources();
schedulingRequest.getResourceSizing().setResources(
getNormalizedResource(resource));{code} Like maybe add a normalize
method to the resource sizing and/or move this operation into the
{{createRMContainer()}} method.
* {{Resources.greaterThan(none)}} probably isn't what you want. You probably
want {{!Resources.isNone()}}.
On [~haibo.chen]'s synchronization concerns, the only thing that concerns me is
the {{FSAppAttempt}} state. I'd have to dig to see if it's really an issue,
though.
> Allow PlacementProcessor to be used with the FairScheduler
> --
>
> Key: YARN-7819
> URL: https://issues.apache.org/jira/browse/YARN-7819
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-7819-YARN-6592.001.patch,
> YARN-7819-YARN-7812.001.patch
>
>
> The FairScheduler needs to implement the
> {{ResourceScheduler#attemptAllocationOnNode}} function for the processor to
> support the FairScheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7739) DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345920#comment-16345920 ] Daniel Templeton commented on YARN-7739: It would be nice to have the test not hard-coded to CS. Would it be too much to ask to parameterize it? You could even leave CS as the only parameter and file a JIRA to add FS. Otherwise, I think it looks good. > DefaultAMSProcessor should properly check customized resource types against > minimum/maximum allocation > -- > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7739.001.patch > > > Currently, YARN RM reject requested resource if memory or vcores are less > than 0 or greater than maximum allocation. We should run the check for > customized resource types as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[ https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345918#comment-16345918 ] Daniel Templeton commented on YARN-7292: I these are still relevant questions and something I'd like to sort out for 3.1. My thoughts: 1) I could go either way on the client v/s server, but I think it's more useful to have on the client side. It makes the code significantly simpler on the server side and allows users to define their own profiles. 2) No, no, and no. They're redundant with what's already specified by resource types. 3) Same question as #1. I vote for client side. 4) If we're supporting server-side profiles, overrides are required in order for the profiles to be useful. If we're going with client-side profiles, then overrides are moot. > Revisit Resource Profile Behavior > - > > Key: YARN-7292 > URL: https://issues.apache.org/jira/browse/YARN-7292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a > couple of resource profile related behaviors might need to be updated: > 1) Configure resource profile in server side or client side: > Currently resource profile can be only configured centrally: > - Advantages: > A given resource profile has a the same meaning in the cluster. It won’t > change when we run different apps in different configurations. A job can run > under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit > is YARN scheduler can potentially do better bin packing. > - Disadvantages: > Hard for applications to add their own resource profiles. > 2) Do we really need mandatory resource profiles such as > minimum/maximum/default? > 3) Should we send resource profile name inside ResourceRequest, or should > client/AM translate it to resource and set it to the existing resource > fields? > 4) Related to above, should we allow resource overrides or client/AM should > send final resource to RM? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7528) Resource types that use units need to be defined at RM level and NM level or when using small units you will overflow max_allocation calculation
[
https://issues.apache.org/jira/browse/YARN-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345503#comment-16345503
]
Daniel Templeton commented on YARN-7528:
If you do not specify a maximum value for a resource, it's set internally to
{{Long.MAX_VALUE}}. When a job is submitted, we check the resource request
against the maximum for the resource, which will require a units conversion.
Hmmm... Now that I write that out loud, it occurs to me that we should be
converting the requested value into the default units, not the other way
around, which would be why you're not seeing an issue. I'll need to look at
the code again. Even if that's the case, you should still be able to cause the
failure by requesting -Dmapreduce.map.resource.gpu=92233720368547k with your
setup above.
> Resource types that use units need to be defined at RM level and NM level or
> when using small units you will overflow max_allocation calculation
>
>
> Key: YARN-7528
> URL: https://issues.apache.org/jira/browse/YARN-7528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, resourcemanager
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
>Priority: Major
>
> When the unit is not defined in the RM, the LONG_MAX default will overflow in
> the conversion step.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7528) Resource types that use units need to be defined at RM level and NM level or when using small units you will overflow max_allocation calculation
[
https://issues.apache.org/jira/browse/YARN-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344290#comment-16344290
]
Daniel Templeton commented on YARN-7528:
Sorry it took me a while to respond. Did you try requesting
{{-Dmapreduce.map.resource.gpu=5k}} with that setup? I don't remember the
details, but I think that if you now ask for a quantity without a unit, it will
assume the resource's default unit. If you force the unit to something larger
than millis, it should expose the issue. As far as I know, the issue should
still exist.
> Resource types that use units need to be defined at RM level and NM level or
> when using small units you will overflow max_allocation calculation
>
>
> Key: YARN-7528
> URL: https://issues.apache.org/jira/browse/YARN-7528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, resourcemanager
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
>Priority: Major
>
> When the unit is not defined in the RM, the LONG_MAX default will overflow in
> the conversion step.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[
https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344252#comment-16344252
]
Daniel Templeton commented on YARN-7556:
During my break I realized that I need to make sure that resource types in
{{maxResources}} don't get set to 0. Otherwise it can screw up the shares
calculations. I'll update the patch for that and your doc issue.
> Fair scheduler configuration should allow resource types in the minResources
> and maxResources properties
>
>
> Key: YARN-7556
> URL: https://issues.apache.org/jira/browse/YARN-7556
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 3.0.0-beta1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7556.001.patch, YARN-7556.002.patch,
> YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch,
> YARN-7556.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329136#comment-16329136 ] Daniel Templeton commented on YARN-7159: I'll add it to my list. I'm just back from being out for a while, so it might take me a couple of days. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch, > YARN-7159.019.patch, YARN-7159.020.patch, YARN-7159.021.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7528) Resource types that use units need to be defined at RM level and NM level or when using small units you will overflow max_allocation calculation
[ https://issues.apache.org/jira/browse/YARN-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318568#comment-16318568 ] Daniel Templeton commented on YARN-7528: Fine with me. > Resource types that use units need to be defined at RM level and NM level or > when using small units you will overflow max_allocation calculation > > > Key: YARN-7528 > URL: https://issues.apache.org/jira/browse/YARN-7528 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation, resourcemanager >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Daniel Templeton > > When the unit is not defined in the RM, the LONG_MAX default will overflow in > the conversion step. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7557) It should be possible to specify resource types in the fair scheduler increment value
[ https://issues.apache.org/jira/browse/YARN-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287931#comment-16287931 ] Daniel Templeton commented on YARN-7557: YARN-7556 is about setting the max and min resources in the *queue* configuration. This is about setting the increment in the *scheduler* configuration. Setting the min and max in the scheduler configuration was part of the initial resource types work, and there is no increment in the queue configuration. > It should be possible to specify resource types in the fair scheduler > increment value > - > > Key: YARN-7557 > URL: https://issues.apache.org/jira/browse/YARN-7557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Gergo Repas >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7624) RM gives YARNFeatureNotEnabledException even when resource profile feature is not enabled
[
https://issues.apache.org/jira/browse/YARN-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283013#comment-16283013
]
Daniel Templeton commented on YARN-7624:
This is against trunk/3.1, yes?
> RM gives YARNFeatureNotEnabledException even when resource profile feature is
> not enabled
> -
>
> Key: YARN-7624
> URL: https://issues.apache.org/jira/browse/YARN-7624
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>
> A single node setup, I haven't enabled resource profile feature. Property
> {{yarn.resourcemanager.resource-profiles.enabled}} was not set. Start yarn,
> launch a job, I got following error message in RM log
> {noformat}
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1822)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:657)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:617)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {noformat}
> this is confusing because I did not enable this feature, why I still get this
> error?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[ https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282133#comment-16282133 ] Daniel Templeton commented on YARN-7119: +1 [~sunilg], wanna do the commit? I'm on vacation still. :) > yarn rmadmin -updateNodeResource should be updated for resource types > - > > Key: YARN-7119 > URL: https://issues.apache.org/jira/browse/YARN-7119 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Attachments: YARN-7119.001.patch, YARN-7119.002.patch, > YARN-7119.002.patch, YARN-7119.003.patch, YARN-7119.004.patch, > YARN-7119.004.patch, YARN-7119.005.patch, YARN-7119.006.patch, > YARN-7119.007.patch, YARN-7119.008.patch, YARN-7119.009.patch, > YARN-7119.010.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280739#comment-16280739 ] Daniel Templeton commented on YARN-7556: Test failures are still unrelated. [~rkanter] or [~wilfreds], any comments? > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[ https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280559#comment-16280559 ] Daniel Templeton commented on YARN-7119: LGTM. Last thing, it would be nice to clean up the checkstyle complaint. +1 after that. > yarn rmadmin -updateNodeResource should be updated for resource types > - > > Key: YARN-7119 > URL: https://issues.apache.org/jira/browse/YARN-7119 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Attachments: YARN-7119.001.patch, YARN-7119.002.patch, > YARN-7119.002.patch, YARN-7119.003.patch, YARN-7119.004.patch, > YARN-7119.004.patch, YARN-7119.005.patch, YARN-7119.006.patch, > YARN-7119.007.patch, YARN-7119.008.patch, YARN-7119.009.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279476#comment-16279476 ] Daniel Templeton commented on YARN-7556: Test failures are all unrelated. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[
https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279185#comment-16279185
]
Daniel Templeton commented on YARN-7119:
Looks good to me, except one tiny issue, which might not even be an issue. In
the javadoc for {{parseResourceValue()}}, you use '>'. The javadoc will be
converted into HTML, and it won't automatically replace that character with
{{>}}. I don't remember if HTML parsers will ignore a closing angle bracket
without an opening angle bracket, but to be safe, I'd either use the HTML code
or find another way to say it.
> yarn rmadmin -updateNodeResource should be updated for resource types
> -
>
> Key: YARN-7119
> URL: https://issues.apache.org/jira/browse/YARN-7119
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Attachments: YARN-7119.001.patch, YARN-7119.002.patch,
> YARN-7119.002.patch, YARN-7119.003.patch, YARN-7119.004.patch,
> YARN-7119.004.patch, YARN-7119.005.patch, YARN-7119.006.patch,
> YARN-7119.007.patch, YARN-7119.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7556: --- Attachment: YARN-7556.006.patch Reposting patch 6 because I think it might have been incomplete. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7556: --- Attachment: (was: YARN-7556.006.patch) > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7556: --- Attachment: YARN-7556.006.patch Whoops. Used the wrong branch. Here's a trunk patch. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch, > YARN-7556.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7556: --- Attachment: YARN-7556.005.patch I reworked the patch to not use -1 as a flag, so there's no longer any risk of things going totally off the rails. To address Wilfred's points: # The fair scheduler docs do say to use memory-mb in the example for maxResources, and the new "Resource Model" page also talks about the way resources work now. # I added support for percentages. It was an intentional omission, but not a good choice. # The decimal thing must be a red herring. The CPU and memory were and still are stored as integers. Allowing decimal values is just lying to the users. The old format still allows the lie, but I see no reason to perpetuate the lie in the new format. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch, YARN-7556.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties
[ https://issues.apache.org/jira/browse/YARN-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277085#comment-16277085 ] Daniel Templeton commented on YARN-7556: The -1's also bother me. I mitigated the risk by keeping them contained. The -1's only exist inside the configuration. Any time you try to extract a resource, the -1's get replaced. I'll take another look, though, to see if there's another way. > Fair scheduler configuration should allow resource types in the minResources > and maxResources properties > > > Key: YARN-7556 > URL: https://issues.apache.org/jira/browse/YARN-7556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-7556.001.patch, YARN-7556.002.patch, > YARN-7556.003.patch, YARN-7556.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7600) Yarn NODE_LOCAL request downgraded to RACK_LOCAL didn't cancel the original NODE_LOCAL request
[ https://issues.apache.org/jira/browse/YARN-7600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275687#comment-16275687 ] Daniel Templeton commented on YARN-7600: [~rkanter], any insights from you? > Yarn NODE_LOCAL request downgraded to RACK_LOCAL didn't cancel the original > NODE_LOCAL request > --- > > Key: YARN-7600 > URL: https://issues.apache.org/jira/browse/YARN-7600 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 2.7.3 >Reporter: wuchang > > I know, when AM making container request, if the request container is > NODE_LOCAL, then AM will also send out RACK_LOCAL and OFF_SWITCH requests. On > the ResourceManager side, if RM successfully assigned a NODE_LOCAL container, > I saw that RM canceled RACK_LOCAL and OFF_SWITCH requests because it was a > duplicated request and did not need to be allocated any more. > However, my problem is that if the **allowRelaxity** for RACK_LOCAL and > OFF_SWITCH is true, NODE_LOCAL request is downgraded to RACK_LOCAL , thus > NODE_LOCAL's request is allocated with RACK_LOCAL locality, then OFF_SWITCH's > duplicated request will be canceled of cource, however I did not see RM > cancel the duplicated NODE_LOCAL request, so, won't it lead to NODE_LOCAL > request still exist and then be scheduled in the next-round scheduling? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7600) Yarn NODE_LOCAL request downgraded to RACK_LOCAL didn't cancel the original NODE_LOCAL request
[ https://issues.apache.org/jira/browse/YARN-7600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275686#comment-16275686 ] Daniel Templeton commented on YARN-7600: We are both out on vacation for December. :) If I get some spare time I'll take a look. > Yarn NODE_LOCAL request downgraded to RACK_LOCAL didn't cancel the original > NODE_LOCAL request > --- > > Key: YARN-7600 > URL: https://issues.apache.org/jira/browse/YARN-7600 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 2.7.3 >Reporter: wuchang > > I know, when AM making container request, if the request container is > NODE_LOCAL, then AM will also send out RACK_LOCAL and OFF_SWITCH requests. On > the ResourceManager side, if RM successfully assigned a NODE_LOCAL container, > I saw that RM canceled RACK_LOCAL and OFF_SWITCH requests because it was a > duplicated request and did not need to be allocated any more. > However, my problem is that if the **allowRelaxity** for RACK_LOCAL and > OFF_SWITCH is true, NODE_LOCAL request is downgraded to RACK_LOCAL , thus > NODE_LOCAL's request is allocated with RACK_LOCAL locality, then OFF_SWITCH's > duplicated request will be canceled of cource, however I did not see RM > cancel the duplicated NODE_LOCAL request, so, won't it lead to NODE_LOCAL > request still exist and then be scheduled in the next-round scheduling? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7589) TestPBImplRecords fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275099#comment-16275099
]
Daniel Templeton commented on YARN-7589:
Thanks for the review and commit, [~jlowe]!
> TestPBImplRecords fails with NullPointerException
> -
>
> Key: YARN-7589
> URL: https://issues.apache.org/jira/browse/YARN-7589
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0, 3.0.1
>Reporter: Jason Lowe
>Assignee: Daniel Templeton
> Fix For: 3.0.0, 3.1.0, 3.0.1
>
> Attachments: YARN-7589.001.patch
>
>
> TestPBImplRecords is failing consistently in trunk:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.413
> s <<< FAILURE! - in org.apache.hadoop.yarn.api.TestPBImplRecords
> [ERROR] org.apache.hadoop.yarn.api.TestPBImplRecords Time elapsed: 0.413 s
> <<< ERROR!
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.api.BasePBImplRecordsTest.generateByNewInstance(BasePBImplRecordsTest.java:151)
> at
> org.apache.hadoop.yarn.api.TestPBImplRecords.setup(TestPBImplRecords.java:371)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.util.resource.ResourceUtils.createResourceTypesArray(ResourceUtils.java:644)
> at
> org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:105)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7589) TestPBImplRecords fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7589:
---
Affects Version/s: 3.0.0
Target Version/s: 3.0.0, 3.1.0, 3.0.1 (was: 3.1.0, 3.0.1)
Fix Version/s: 3.0.0
Pulled back into 3.0.0 as well since YARN-7541 is there.
> TestPBImplRecords fails with NullPointerException
> -
>
> Key: YARN-7589
> URL: https://issues.apache.org/jira/browse/YARN-7589
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0, 3.0.1
>Reporter: Jason Lowe
>Assignee: Daniel Templeton
> Fix For: 3.0.0, 3.1.0, 3.0.1
>
> Attachments: YARN-7589.001.patch
>
>
> TestPBImplRecords is failing consistently in trunk:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.413
> s <<< FAILURE! - in org.apache.hadoop.yarn.api.TestPBImplRecords
> [ERROR] org.apache.hadoop.yarn.api.TestPBImplRecords Time elapsed: 0.413 s
> <<< ERROR!
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.api.BasePBImplRecordsTest.generateByNewInstance(BasePBImplRecordsTest.java:151)
> at
> org.apache.hadoop.yarn.api.TestPBImplRecords.setup(TestPBImplRecords.java:371)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.util.resource.ResourceUtils.createResourceTypesArray(ResourceUtils.java:644)
> at
> org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:105)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7589) TestPBImplRecords fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274932#comment-16274932
]
Daniel Templeton commented on YARN-7589:
Findbugs issue is unrelated.
> TestPBImplRecords fails with NullPointerException
> -
>
> Key: YARN-7589
> URL: https://issues.apache.org/jira/browse/YARN-7589
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1
>Reporter: Jason Lowe
>Assignee: Daniel Templeton
> Attachments: YARN-7589.001.patch
>
>
> TestPBImplRecords is failing consistently in trunk:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.413
> s <<< FAILURE! - in org.apache.hadoop.yarn.api.TestPBImplRecords
> [ERROR] org.apache.hadoop.yarn.api.TestPBImplRecords Time elapsed: 0.413 s
> <<< ERROR!
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.api.BasePBImplRecordsTest.generateByNewInstance(BasePBImplRecordsTest.java:151)
> at
> org.apache.hadoop.yarn.api.TestPBImplRecords.setup(TestPBImplRecords.java:371)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.util.resource.ResourceUtils.createResourceTypesArray(ResourceUtils.java:644)
> at
> org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:105)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7589) TestPBImplRecords fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7589:
---
Attachment: YARN-7589.001.patch
Silly oversight on my part. This patch fixes the issue but does not separate
out the tests as [~jlowe] suggested on YARN-7541. I'm on vacation, so let's
file another JIRA to split the test.
> TestPBImplRecords fails with NullPointerException
> -
>
> Key: YARN-7589
> URL: https://issues.apache.org/jira/browse/YARN-7589
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1
>Reporter: Jason Lowe
> Attachments: YARN-7589.001.patch
>
>
> TestPBImplRecords is failing consistently in trunk:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.413
> s <<< FAILURE! - in org.apache.hadoop.yarn.api.TestPBImplRecords
> [ERROR] org.apache.hadoop.yarn.api.TestPBImplRecords Time elapsed: 0.413 s
> <<< ERROR!
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.api.BasePBImplRecordsTest.generateByNewInstance(BasePBImplRecordsTest.java:151)
> at
> org.apache.hadoop.yarn.api.TestPBImplRecords.setup(TestPBImplRecords.java:371)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.util.resource.ResourceUtils.createResourceTypesArray(ResourceUtils.java:644)
> at
> org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:105)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7541) Node updates don't update the maximum cluster capability for resources other than CPU and memory
[
https://issues.apache.org/jira/browse/YARN-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273967#comment-16273967
]
Daniel Templeton commented on YARN-7541:
Weird. That test didn't fail in the pre-commits. I'll try to get to it
tomorrow morning.
> Node updates don't update the maximum cluster capability for resources other
> than CPU and memory
>
>
> Key: YARN-7541
> URL: https://issues.apache.org/jira/browse/YARN-7541
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7541.001.patch, YARN-7541.002.patch,
> YARN-7541.003.patch, YARN-7541.004.patch, YARN-7541.005.patch,
> YARN-7541.006.patch, YARN-7541.branch-3.0.001.patch
>
>
> When I submit an MR job that asks for too much memory or CPU for the map or
> reduce, the AM will fail because it recognizes that the request is too large.
> With any other resources, however, the resource requests will instead be
> made and remain pending forever. Looks like we forgot to update the code
> that tracks the maximum container allocation in {{ClusterNodeTracker}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7589) TestPBImplRecords fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273432#comment-16273432
]
Daniel Templeton commented on YARN-7589:
I will take a look when I get a chance.
> TestPBImplRecords fails with NullPointerException
> -
>
> Key: YARN-7589
> URL: https://issues.apache.org/jira/browse/YARN-7589
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1
>Reporter: Jason Lowe
>
> TestPBImplRecords is failing consistently in trunk:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.413
> s <<< FAILURE! - in org.apache.hadoop.yarn.api.TestPBImplRecords
> [ERROR] org.apache.hadoop.yarn.api.TestPBImplRecords Time elapsed: 0.413 s
> <<< ERROR!
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.api.BasePBImplRecordsTest.generateByNewInstance(BasePBImplRecordsTest.java:151)
> at
> org.apache.hadoop.yarn.api.TestPBImplRecords.setup(TestPBImplRecords.java:371)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.util.resource.ResourceUtils.createResourceTypesArray(ResourceUtils.java:644)
> at
> org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:105)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
