[jira] [Commented] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718561#comment-16718561
]
Akhil PB commented on YARN-9089:
[~eyang] Couple of comments.
# We don't need {{termLink: DS.attr('string')}} in
{{models/yarn-component-instance.js}}. Since termLink is a computed property
bases on node and containerId.
# {{termLink: Ember.computed('termLink', function() }} should be {{termLink:
Ember.computed('node', 'containerId', function() }}
# Please use {{ window.location.protocol }} instead of {{ location.protocol }}
(eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718561#comment-16718561
]
Akhil PB edited comment on YARN-9089 at 12/12/18 7:41 AM:
--
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed property
bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{window.location.protocol}} instead of {{ location.protocol }}
(eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
was (Author: akhilpb):
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed property
bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{window.location.protocol}} instead of {{ location.protocol }}
(eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718561#comment-16718561
]
Akhil PB edited comment on YARN-9089 at 12/12/18 7:39 AM:
--
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed
property bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{ window.location.protocol }} instead of \{{ location.protocol
}} (eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
was (Author: akhilpb):
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed
property bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{ window.location.protocol }} instead of \{{ location.protocol
}} (eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718561#comment-16718561
]
Akhil PB edited comment on YARN-9089 at 12/12/18 7:39 AM:
--
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed property
bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{window.location.protocol}} instead of {{ location.protocol }}
(eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
was (Author: akhilpb):
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed
property bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{ window.location.protocol }} instead of \{{ location.protocol
}} (eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718561#comment-16718561
]
Akhil PB edited comment on YARN-9089 at 12/12/18 7:38 AM:
--
[~eyang] Couple of comments.
# We don't need
{noformat}
termLink: DS.attr('string')
{noformat}
in {{models/yarn-component-instance.js}}. Since termLink is a computed
property bases on node and containerId.
# The line
{noformat}
termLink: Ember.computed('termLink', function ()
{noformat}
should be
{noformat}
termLink: Ember.computed('node', 'containerId', function()
{noformat}
# Please use {{ window.location.protocol }} instead of \{{ location.protocol
}} (eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
was (Author: akhilpb):
[~eyang] Couple of comments.
# We don't need {{termLink: DS.attr('string')}} in
{{models/yarn-component-instance.js}}. Since termLink is a computed property
bases on node and containerId.
# {{termLink: Ember.computed('termLink', function() }} should be {{termLink:
Ember.computed('node', 'containerId', function() }}
# Please use {{ window.location.protocol }} instead of {{ location.protocol }}
(eventhough both are same) to show clearly that we are accessing browser
protocal property, not some local variable or something.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9093) Remove commented code block from the beginning of TestDefaultContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-9093: Assignee: Vidura Bhathiya Mudalige > Remove commented code block from the beginning of TestDefaultContainerExecutor > -- > > Key: YARN-9093 > URL: https://issues.apache.org/jira/browse/YARN-9093 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Vidura Bhathiya Mudalige >Priority: Trivial > Labels: newbie, newbie++ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9114) [UI2] log service redirect url should also support user name
[ https://issues.apache.org/jira/browse/YARN-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-9114: --- Attachment: YARN-9114.001.patch > [UI2] log service redirect url should also support user name > > > Key: YARN-9114 > URL: https://issues.apache.org/jira/browse/YARN-9114 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp, yarn-ui-v2 >Reporter: Sunil Govindan >Assignee: Akhil PB >Priority: Major > Attachments: YARN-9114.001.patch > > > Add username to yarn-log-service end point -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9093) Remove commented code block from the beginning of TestDefaultContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718535#comment-16718535 ] Szilard Nemeth commented on YARN-9093: -- Hi [~vbmudalige]! Sure, assigned this to you. Btw, you should have thr right to assign jiras to yourself. IIRC, I got it from one of the PMCs (not sure if committers can give you that right). [~haibochen]: can you please confirm? Thanks! > Remove commented code block from the beginning of TestDefaultContainerExecutor > -- > > Key: YARN-9093 > URL: https://issues.apache.org/jira/browse/YARN-9093 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Vidura Bhathiya Mudalige >Priority: Trivial > Labels: newbie, newbie++ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9114) [UI2] log service redirect url should also support user name
Sunil Govindan created YARN-9114: Summary: [UI2] log service redirect url should also support user name Key: YARN-9114 URL: https://issues.apache.org/jira/browse/YARN-9114 Project: Hadoop YARN Issue Type: Bug Components: webapp, yarn-ui-v2 Reporter: Sunil Govindan Assignee: Akhil PB Add username to yarn-log-service end point -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6416) SIGNAL_CMD argument number is wrong
[
https://issues.apache.org/jira/browse/YARN-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vidura Bhathiya Mudalige reassigned YARN-6416:
--
Assignee: Vidura Bhathiya Mudalige
> SIGNAL_CMD argument number is wrong
> ---
>
> Key: YARN-6416
> URL: https://issues.apache.org/jira/browse/YARN-6416
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Vidura Bhathiya Mudalige
>Priority: Minor
> Labels: newbie
>
> Yarn application signal command has two arguments, so the number below should
> be 2 I think.
> {code}
> opts.getOption(SIGNAL_CMD).setArgs(3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9095) Removed Unused field from Resource: NUM_MANDATORY_RESOURCES
[ https://issues.apache.org/jira/browse/YARN-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vidura Bhathiya Mudalige reassigned YARN-9095: -- Assignee: Vidura Bhathiya Mudalige > Removed Unused field from Resource: NUM_MANDATORY_RESOURCES > --- > > Key: YARN-9095 > URL: https://issues.apache.org/jira/browse/YARN-9095 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Vidura Bhathiya Mudalige >Priority: Trivial > Labels: newbie, newbie++ > > I suppose this constant remained in the code from historical reasons, but > this is not used anymore so it could be removed. > This field is especially confusing for new readers, as ResourceInformation > now has a field named MANDATORY_RESOURCES and this map contains not only > memory and vcores but GPU and FPGA as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9095) Removed Unused field from Resource: NUM_MANDATORY_RESOURCES
[ https://issues.apache.org/jira/browse/YARN-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718516#comment-16718516 ] ASF GitHub Bot commented on YARN-9095: -- GitHub user vbmudalige opened a pull request: https://github.com/apache/hadoop/pull/445 YARN-9095. Removed Unused field from Resource: NUM_MANDATORY_RESOURCES You can merge this pull request into a Git repository by running: $ git pull https://github.com/vbmudalige/hadoop YARN-9095 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/445.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #445 commit a1cf1a6aa1d8ea93b2efef277878b813c4551e6c Author: Vidura Mudalige Date: 2018-12-12T06:40:02Z YARN-9095. Removed Unused field from Resource: NUM_MANDATORY_RESOURCES > Removed Unused field from Resource: NUM_MANDATORY_RESOURCES > --- > > Key: YARN-9095 > URL: https://issues.apache.org/jira/browse/YARN-9095 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Trivial > Labels: newbie, newbie++ > > I suppose this constant remained in the code from historical reasons, but > this is not used anymore so it could be removed. > This field is especially confusing for new readers, as ResourceInformation > now has a field named MANDATORY_RESOURCES and this map contains not only > memory and vcores but GPU and FPGA as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9093) Remove commented code block from the beginning of TestDefaultContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718500#comment-16718500 ] Vidura Bhathiya Mudalige commented on YARN-9093: [~snemeth], can you please assign this Jira to me? > Remove commented code block from the beginning of TestDefaultContainerExecutor > -- > > Key: YARN-9093 > URL: https://issues.apache.org/jira/browse/YARN-9093 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Trivial > Labels: newbie, newbie++ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9093) Remove commented code block from the beginning of TestDefaultContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718499#comment-16718499 ] ASF GitHub Bot commented on YARN-9093: -- GitHub user vbmudalige opened a pull request: https://github.com/apache/hadoop/pull/444 YARN-9093. Remove commented code block from the beginning of TestDefa… …ultContainerExecutor You can merge this pull request into a Git repository by running: $ git pull https://github.com/vbmudalige/hadoop YARN-9093 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/444.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #444 commit eba9ac79e9c5db4d0bce57d7d7afae8446a5a423 Author: Vidura Mudalige Date: 2018-12-12T05:59:09Z YARN-9093. Remove commented code block from the beginning of TestDefaultContainerExecutor > Remove commented code block from the beginning of TestDefaultContainerExecutor > -- > > Key: YARN-9093 > URL: https://issues.apache.org/jira/browse/YARN-9093 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Trivial > Labels: newbie, newbie++ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9112) [Submarine] Support polling applicationId when it's not ready in cluster
[
https://issues.apache.org/jira/browse/YARN-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718471#comment-16718471
]
Hadoop QA commented on YARN-9112:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951453/YARN-9112-trunk.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 24aea8b1a86a 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb55e52 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22841/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22841/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Created] (YARN-9113) [Submarine] Add proper shutdown hook when user interrupts the client
Zhankun Tang created YARN-9113: -- Summary: [Submarine] Add proper shutdown hook when user interrupts the client Key: YARN-9113 URL: https://issues.apache.org/jira/browse/YARN-9113 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhankun Tang Assignee: Zhankun Tang When the job run with "wait_job_finish" option and then the user "ctrl+c" the job. The close of the service client is missing. We can add a shutdown hook in this scenario to avoid this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9112) [Submarine] Support polling applicationId when it's not ready in cluster
[
https://issues.apache.org/jira/browse/YARN-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718431#comment-16718431
]
Zhankun Tang commented on YARN-9112:
[~wangda] , please help to review. Thanks.
> [Submarine] Support polling applicationId when it's not ready in cluster
>
>
> Key: YARN-9112
> URL: https://issues.apache.org/jira/browse/YARN-9112
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9112-trunk.001.patch
>
>
> This could happen when an application is not ready in the cluster. A polling
> for the application Id is needed.
> {code:java}
> 18/12/11 22:39:58 INFO client.ApiServiceClient: Application ID:
> application_1532131617202_0063
> Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
> Can't get application id for Service tensorboard-service
> at
> org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.submitJob(YarnServiceJobSubmitter.java:618)
> at
> org.apache.hadoop.yarn.submarine.client.cli.RunJobCli.run(RunJobCli.java:241)
> at org.apache.hadoop.yarn.submarine.client.cli.Cli.main(Cli.java:91)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9112) [Submarine] Support polling applicationId when it's not ready in cluster
[
https://issues.apache.org/jira/browse/YARN-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-9112:
---
Attachment: YARN-9112-trunk.001.patch
> [Submarine] Support polling applicationId when it's not ready in cluster
>
>
> Key: YARN-9112
> URL: https://issues.apache.org/jira/browse/YARN-9112
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9112-trunk.001.patch
>
>
> This could happen when an application is not ready in the cluster. A polling
> for the application Id is needed.
> {code:java}
> 18/12/11 22:39:58 INFO client.ApiServiceClient: Application ID:
> application_1532131617202_0063
> Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
> Can't get application id for Service tensorboard-service
> at
> org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.submitJob(YarnServiceJobSubmitter.java:618)
> at
> org.apache.hadoop.yarn.submarine.client.cli.RunJobCli.run(RunJobCli.java:241)
> at org.apache.hadoop.yarn.submarine.client.cli.Cli.main(Cli.java:91)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9112) [Submarine] Support polling applicationId when it's not ready in cluster
Zhankun Tang created YARN-9112:
--
Summary: [Submarine] Support polling applicationId when it's not
ready in cluster
Key: YARN-9112
URL: https://issues.apache.org/jira/browse/YARN-9112
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Zhankun Tang
Assignee: Zhankun Tang
This could happen when an application is not ready in the cluster. A polling
for the application Id is needed.
{code:java}
18/12/11 22:39:58 INFO client.ApiServiceClient: Application ID:
application_1532131617202_0063
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
Can't get application id for Service tensorboard-service
at
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.submitJob(YarnServiceJobSubmitter.java:618)
at
org.apache.hadoop.yarn.submarine.client.cli.RunJobCli.run(RunJobCli.java:241)
at org.apache.hadoop.yarn.submarine.client.cli.Cli.main(Cli.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7291) Better input parsing for resource in allocation file
[
https://issues.apache.org/jira/browse/YARN-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718301#comment-16718301
]
Hadoop QA commented on YARN-7291:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 15s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerConfiguration
|
| | hadoop.yarn.server.resourcemanager.TestAppManagerWithFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestApplicationMasterServiceWithFS
|
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-7291 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951367/YARN-7291.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7d32dc7c069c 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb55e52 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22840/artifact/out/patch-unit-hadoop-yarn-project_had
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718234#comment-16718234
]
Wangda Tan commented on YARN-8714:
--
[~tangzhankun], sounds like a plan, but let's try to solve the issue on service
side early if possible. Could u file a new Jira and work on that when you get
chance?
+1 to the latest patch.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch,
> YARN-8714-trunk.002.patch, YARN-8714-trunk.003.patch,
> YARN-8714-trunk.004.patch, YARN-8714-trunk.005.patch,
> YARN-8714-trunk.006.patch, YARN-8714-trunk.007.patch,
> YARN-8714-trunk.008.patch, YARN-8714-trunk.009.patch,
> YARN-8714-trunk.010.patch
>
>
> See
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
> {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[ https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718145#comment-16718145 ] Eric Yang commented on YARN-9089: - [~akhilpb] Please review patch 002. Thanks > Add Terminal Link to Service component instance page for UI2 > > > Key: YARN-9089 > URL: https://issues.apache.org/jira/browse/YARN-9089 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Affects Versions: 3.3.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9089.001.patch, YARN-9089.002.patch > > > In UI2, Service > Component > Component Instance uses Timeline server to > aggregate information about Service component instance. Timeline server does > not have the full information like the port number of the node manager, or > the web protocol used by the node manager. It requires some changes to > aggregate node manager information into Timeline server in order to compute > the Terminal link. For reducing the scope of YARN-8914, it is better file > this as a separate task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7720) Race condition between second app attempt and UAM timeout when first attempt node is down
[
https://issues.apache.org/jira/browse/YARN-7720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718116#comment-16718116
]
Hadoop QA commented on YARN-7720:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 8s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-7720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950247/YARN-7720.v2.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux de01599b1c00 4.4.0-138-generic #164-Ubuntu SMP Tue
[jira] [Created] (YARN-9111) NM crashes because Fair scheduler promotes a container that has not been pulled by AM
Haibo Chen created YARN-9111:
Summary: NM crashes because Fair scheduler promotes a container
that has not been pulled by AM
Key: YARN-9111
URL: https://issues.apache.org/jira/browse/YARN-9111
Project: Hadoop YARN
Issue Type: Sub-task
Components: fairscheduler, nodemanager
Affects Versions: YARN-1011
Reporter: Haibo Chen
{code:java}
2018-10-19 22:34:35,052 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
Error in dispatcher thread
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerTokenIdentifier(BuilderUtils.java:323)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.handle(ContainerManagerImpl.java:1649)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.handle(ContainerManagerImpl.java:185)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:748)
2018-10-19 22:34:35,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Exiting, bbye..
2018-10-19 22:34:35,059 DEBUG org.apache.hadoop.service.AbstractService:
Service: NodeManager entered state STOPPED{code}
When a container is allocated by RM to an application, its container token is
not generated until the AM pulls that container from RM.
However, it the scheduler decides to promote that container before it is pulled
by the AM, it does not have container token to work with.
The current code does not update/generate the container token as such. When
container promotion is sent to NM to process, the NM crashes on NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9110) Fair Scheduler promotion does not update container execution type when the application is killed
Haibo Chen created YARN-9110: Summary: Fair Scheduler promotion does not update container execution type when the application is killed Key: YARN-9110 URL: https://issues.apache.org/jira/browse/YARN-9110 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Affects Versions: YARN-1011 Reporter: Haibo Chen When the application that a container belongs to gets killed right before the container is promoted, the resource booking keeping is updated to reflect the promotion, or, but the container token won't be updated.Hence when the container is released, it is seen as an OPPORTUNISTIC container. We need to make the container promotion, including resource bookkeeping and token update atomic to avoid this problem -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718099#comment-16718099 ] Giovanni Matteo Fumarola commented on YARN-9107: Thanks [~busbey] . I will work with [~briangru] to validate that this change is not breaking anything in Linux side. > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718110#comment-16718110 ] Hudson commented on YARN-9008: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15594 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15594/]) YARN-9008. Extend YARN distributed shell with file localization feature. (haibochen: rev fb55e5201e5b2ff40e1b757a9c5bf23c5d8aec93) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/a.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/b.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > Extend YARN distributed shell with file localization feature > > > Key: YARN-9008 > URL: https://issues.apache.org/jira/browse/YARN-9008 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9008-001.patch, YARN-9008-002.patch, > YARN-9008-003.patch, YARN-9008-004.patch, YARN-9008-005.patch, > YARN-9008-006.patch, YARN-9008-007.patch > > > YARN distributed shell is a very handy tool to test various features of YARN. > However, it lacks support for file localization - that is, you define files > in the command line that you wish to be localized remotely. This can be > extremely useful in certain scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718097#comment-16718097 ] Sean Busbey commented on YARN-9107: --- Building on windows and running on windows are two mostly-independent things. I agree that it would be nice to continue building on Windows. Every review is a relatively large time investment for me. I haven't issued any -1 so folks are free to carry on without waiting for me; I'm just setting expectations. > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718096#comment-16718096 ] Haibo Chen commented on YARN-9008: -- +1 on the latest patch. Checking it in shortly. > Extend YARN distributed shell with file localization feature > > > Key: YARN-9008 > URL: https://issues.apache.org/jira/browse/YARN-9008 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9008-001.patch, YARN-9008-002.patch, > YARN-9008-003.patch, YARN-9008-004.patch, YARN-9008-005.patch, > YARN-9008-006.patch, YARN-9008-007.patch > > > YARN distributed shell is a very handy tool to test various features of YARN. > However, it lacks support for file localization - that is, you define files > in the command line that you wish to be localized remotely. This can be > extremely useful in certain scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718082#comment-16718082 ] Brian Grunkemeyer commented on YARN-9107: - My team is running Hadoop on Windows on hundreds of thousands of machines, and we build Hadoop 2.7.x on Windows today. With the 3.2 branch, there is still a section in BUILDING.txt describing how to build Hadoop on Windows. It would be nice to continue building on Windows. The changes here should be easy to review. This shouldn't be a dramatic time investment. > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9109) DelegationTokenRenewer not resilient to token classload problems
Steve Loughran created YARN-9109:
Summary: DelegationTokenRenewer not resilient to token classload
problems
Key: YARN-9109
URL: https://issues.apache.org/jira/browse/YARN-9109
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 3.1.2
Reporter: Steve Loughran
The DelegationTokenRenewer can't handle the situation of: token implementation
class cannot be instantiated.
This is because {{Token.decodeIdentifier()}} throws an RTE on failure to
instantiate, and these aren't caught.
Appears to surfaces in the client: with an error about "Timer Already
Cancelled". In the logs, you see this
{code}
Exception: java.lang.NoClassDefFoundError thrown from the
UncaughtExceptionHandler in thread "Timer-5"
{code}
At least, I'm assuming this is the cause Relates to HADOOP-14556
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9109) DelegationTokenRenewer not resilient to token classload problems
[
https://issues.apache.org/jira/browse/YARN-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718016#comment-16718016
]
Steve Loughran commented on YARN-9109:
--
Server
{code}
2018-12-11 16:32:17,429 [main] ERROR tools.DistCp (DistCp.java:run(167)) -
Exception encountered
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to
submit application_1543889597027_0032 to YARN : Timer already cancelled.
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:345)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:251)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1576)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1573)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1573)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:206)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1543889597027_0032 to YARN : Timer already cancelled.
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:299)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:330)
{code}
> DelegationTokenRenewer not resilient to token classload problems
>
>
> Key: YARN-9109
> URL: https://issues.apache.org/jira/browse/YARN-9109
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.2
>Reporter: Steve Loughran
>Priority: Minor
>
> The DelegationTokenRenewer can't handle the situation of: token
> implementation class cannot be instantiated.
> This is because {{Token.decodeIdentifier()}} throws an RTE on failure to
> instantiate, and these aren't caught.
> Appears to surfaces in the client: with an error about "Timer Already
> Cancelled". In the logs, you see this
> {code}
> Exception: java.lang.NoClassDefFoundError thrown from the
> UncaughtExceptionHandler in thread "Timer-5"
> {code}
> At least, I'm assuming this is the cause Relates to HADOOP-14556
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718000#comment-16718000 ] Sean Busbey commented on YARN-9107: --- Do we still support building on windows? it'll take me a bit to find time to review. > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with file localization feature
[
https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717958#comment-16717958
]
Hadoop QA commented on YARN-9008:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 5s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9008 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951404/YARN-9008-007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 73741b42f22e 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 202926a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22838/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22838/testReport/ |
| Max. process+thread count | 618 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
|
| Co
[jira] [Commented] (YARN-9051) Integrate multiple CustomResourceTypesConfigurationProvider implementations into one
[
https://issues.apache.org/jira/browse/YARN-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717884#comment-16717884
]
Hudson commented on YARN-9051:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15593 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15593/])
YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider
(haibochen: rev 881230da21cc0e7d48fe2928fa999cc431000f16)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/fpga/TestFpgaResourceHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppCustomResourceTypes.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/fairscheduler/TestRMWebServicesFairSchedulerCustomResourceTypes.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/CustomResourceTypesConfigurationProvider.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
* (delete)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/fairscheduler/CustomResourceTypesConfigurationProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsCustomResourceTypes.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCustomResourceTypesCommons.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
> Integrate multiple CustomResourceTypesConfigurationProvider implementations
> into one
>
>
> Key: YARN-9051
> URL: https://issues.apache.org/jira/browse/YARN-9051
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9051.001.patch, YARN-9051.002.patch,
> YARN-9051.003.patch
>
>
> CustomResourceTypesConfigurationProvider (extends LocalConfigurationProvider)
> has 5 implementations on trunk nowadays.
> These could be integrated into 1 common class.
> Also,
> {{org.apache.hadoop.yarn.util.resource.TestResourceUtils#addNewTypesToResources}}
> has similar functionality so this can be considered as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9047) FairScheduler: default resource calculator is not resource type aware
[
https://issues.apache.org/jira/browse/YARN-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717874#comment-16717874
]
Szilard Nemeth commented on YARN-9047:
--
Hmm jenkins said differently. I don't get how this testcase passed for me when
I ran all the YARN tests on my loval machine. Anway, I checked the code briefly
and I'm on the right track to point out what is this. I will make an update
here later when I can describe the issue more
> FairScheduler: default resource calculator is not resource type aware
> -
>
> Key: YARN-9047
> URL: https://issues.apache.org/jira/browse/YARN-9047
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Wilfred Spiegelenburg
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9047.001.patch, YARN-9047.002.patch,
> YARN-9047.003.patch
>
>
> The FairScheduler#getResourceCalculator always returns the default resource
> calculator. The default calculator is not resource type aware and should only
> be used if there are no resource types configured.
> We need to make sure that in we the direct hard code reference to
> {{RESOURCE_CALCULATOR}} is either safe to use in all cases or is not used in
> the scheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9051) Integrate multiple CustomResourceTypesConfigurationProvider implementations into one
[
https://issues.apache.org/jira/browse/YARN-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717858#comment-16717858
]
Haibo Chen commented on YARN-9051:
--
The unit test failures and license issue are unrelated. +1 on the latest patch
and checking it in shortly.
> Integrate multiple CustomResourceTypesConfigurationProvider implementations
> into one
>
>
> Key: YARN-9051
> URL: https://issues.apache.org/jira/browse/YARN-9051
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-9051.001.patch, YARN-9051.002.patch,
> YARN-9051.003.patch
>
>
> CustomResourceTypesConfigurationProvider (extends LocalConfigurationProvider)
> has 5 implementations on trunk nowadays.
> These could be integrated into 1 common class.
> Also,
> {{org.apache.hadoop.yarn.util.resource.TestResourceUtils#addNewTypesToResources}}
> has similar functionality so this can be considered as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717851#comment-16717851 ] Giovanni Matteo Fumarola commented on YARN-9107: Thanks [~briangru] for the patch. From my side it seems a valid changes for a Windows environments. I am not sure for a Linux environments. cc. [~busbey] since he added those classes in HADOOP-14089. > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9107) Jar validation bash scripts don't work on Windows due to platform differences (colons in paths, \r\n)
[ https://issues.apache.org/jira/browse/YARN-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned YARN-9107: - Assignee: Brian Grunkemeyer > Jar validation bash scripts don't work on Windows due to platform differences > (colons in paths, \r\n) > - > > Key: YARN-9107 > URL: https://issues.apache.org/jira/browse/YARN-9107 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.2.0, 3.3.0 > Environment: Windows 10 > Visual Studio 2017 >Reporter: Brian Grunkemeyer >Assignee: Brian Grunkemeyer >Priority: Blocker > Labels: build, newbie, windows > Attachments: YARN-9107.v1.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Building Hadoop fails on Windows due to a few shell scripts that make invalid > assumptions: > 1) Colon shouldn't be used to separate multiple paths in command line > parameters. Colons occur in Windows paths. > 2) Shell scripts that rely on running external tools need to deal with > carriage return - line feed differences (lines ending in \r\n, not just \n) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9008) Extend YARN distributed shell with file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9008: --- Attachment: YARN-9008-007.patch > Extend YARN distributed shell with file localization feature > > > Key: YARN-9008 > URL: https://issues.apache.org/jira/browse/YARN-9008 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9008-001.patch, YARN-9008-002.patch, > YARN-9008-003.patch, YARN-9008-004.patch, YARN-9008-005.patch, > YARN-9008-006.patch, YARN-9008-007.patch > > > YARN distributed shell is a very handy tool to test various features of YARN. > However, it lacks support for file localization - that is, you define files > in the command line that you wish to be localized remotely. This can be > extremely useful in certain scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717778#comment-16717778 ] Peter Bacsko commented on YARN-9008: Minor checkstyle fix in v7. > Extend YARN distributed shell with file localization feature > > > Key: YARN-9008 > URL: https://issues.apache.org/jira/browse/YARN-9008 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9008-001.patch, YARN-9008-002.patch, > YARN-9008-003.patch, YARN-9008-004.patch, YARN-9008-005.patch, > YARN-9008-006.patch, YARN-9008-007.patch > > > YARN distributed shell is a very handy tool to test various features of YARN. > However, it lacks support for file localization - that is, you define files > in the command line that you wish to be localized remotely. This can be > extremely useful in certain scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717723#comment-16717723
]
Hadoop QA commented on YARN-9089:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9089 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951389/YARN-9089.002.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 488ccf631436 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 202926a |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22836/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch, YARN-9089.002.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9032) Support sh shell for interactive shell
[
https://issues.apache.org/jira/browse/YARN-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717702#comment-16717702
]
Hadoop QA commented on YARN-9032:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-9032 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951396/YARN-9032.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22837/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support sh shell for interactive shell
> --
>
> Key: YARN-9032
> URL: https://issues.apache.org/jira/browse/YARN-9032
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-9032.001.patch, YARN-9032.002.patch
>
>
> CLI can supply second parameter such as:
> yarn container -shell [container-id] sh
> This is supposed to launch sh instead of bash. This is not fully working yet
> in YARN-8778 patch and some organization of code clean up to support
> non-docker container in YARN-8962 making this change in YARN-8778 difficult.
> Hence, I file this as a separate sub-task to close the gap.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9032) Support sh shell for interactive shell
[ https://issues.apache.org/jira/browse/YARN-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-9032: Attachment: YARN-9032.002.patch > Support sh shell for interactive shell > -- > > Key: YARN-9032 > URL: https://issues.apache.org/jira/browse/YARN-9032 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-9032.001.patch, YARN-9032.002.patch > > > CLI can supply second parameter such as: > yarn container -shell [container-id] sh > This is supposed to launch sh instead of bash. This is not fully working yet > in YARN-8778 patch and some organization of code clean up to support > non-docker container in YARN-8962 making this change in YARN-8778 difficult. > Hence, I file this as a separate sub-task to close the gap. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[ https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-9089: Attachment: YARN-9089.002.patch > Add Terminal Link to Service component instance page for UI2 > > > Key: YARN-9089 > URL: https://issues.apache.org/jira/browse/YARN-9089 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Affects Versions: 3.3.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9089.001.patch, YARN-9089.002.patch > > > In UI2, Service > Component > Component Instance uses Timeline server to > aggregate information about Service component instance. Timeline server does > not have the full information like the port number of the node manager, or > the web protocol used by the node manager. It requires some changes to > aggregate node manager information into Timeline server in order to compute > the Terminal link. For reducing the scope of YARN-8914, it is better file > this as a separate task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[ https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717637#comment-16717637 ] Eric Yang commented on YARN-9089: - [~akhilpb] Thanks for the hints. Patch 002 is revised to inject node manager port to ENV config to reduce the ajax call in model. > Add Terminal Link to Service component instance page for UI2 > > > Key: YARN-9089 > URL: https://issues.apache.org/jira/browse/YARN-9089 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Affects Versions: 3.3.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9089.001.patch, YARN-9089.002.patch > > > In UI2, Service > Component > Component Instance uses Timeline server to > aggregate information about Service component instance. Timeline server does > not have the full information like the port number of the node manager, or > the web protocol used by the node manager. It requires some changes to > aggregate node manager information into Timeline server in order to compute > the Terminal link. For reducing the scope of YARN-8914, it is better file > this as a separate task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715796#comment-16715796
]
Eric Yang edited comment on YARN-9089 at 12/11/18 5:08 PM:
---
[~akhilpb] Thank you for the review. Agree that models are not meant for code
execution. The code will be revised. Is the intend to use loader.js to set
ENV.[namespace]=[ajax] value for YARN configuration? Is there a convention to
follow? A example might help with the coding style. Thanks.
{quote}Do we need this code? Since you are adding userInfo to
yarn-component-instance/info route, it should be available in info.hbs page in
model object. I am not sure passing params to outlet works or not in
ember.{quote}
You are right. The parameter passing can be removed. Thanks
was (Author: eyang):
[~akhilpb] Thank you for the review. Agree that models are not meant for code
execution. The code will be revised. Is the intend to use loader.js to set
ENV.[namespace]=[ajax] value for YARN configuration? Is there a convention to
follow? A example might help with the coding style. Thanks.
{quote}Do we need this code? Since you are adding userInfo to
yarn-component-instance/info route, it should be available in info.hbs page in
model object. I am not sure passing params to outlet works or not in
ember.\{quote}
You are right. The parameter passing can be removed. Thanks
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7746) Fix PlacementProcessor to support app priority
[ https://issues.apache.org/jira/browse/YARN-7746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717433#comment-16717433 ] Weiwei Yang commented on YARN-7746: --- Hi [~maniraj...@gmail.com] I was trying to get some more info about the motivation, because I was not in the discussion when this was created. Ping [~asuresh], since he created this task. > Fix PlacementProcessor to support app priority > -- > > Key: YARN-7746 > URL: https://issues.apache.org/jira/browse/YARN-7746 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Manikandan R >Priority: Major > Attachments: YARN-7746.001.patch, YARN-7746.002.patch > > > The Threadpools used in the Processor should be modified to take a priority > blocking queue that respects application priority. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
[ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717432#comment-16717432 ] Weiwei Yang commented on YARN-8925: --- Sure thanks [~Tao Yang]. That is correct, they can co-exist on one cluster. I will take a look this week. Thanks for the updates. > Updating distributed node attributes only when necessary > > > Key: YARN-8925 > URL: https://issues.apache.org/jira/browse/YARN-8925 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: performance > Attachments: YARN-8925.001.patch, YARN-8925.002.patch, > YARN-8925.003.patch, YARN-8925.004.patch, YARN-8925.005.patch, > YARN-8925.006.patch, YARN-8925.007.patch, YARN-8925.008.patch, > YARN-8925.009.patch, YARN-8925.010.patch > > > Currently if distributed node attributes exist, even though there is no > change, updating for distributed node attributes will happen in every > heartbeat between NM and RM. Updating process will hold > NodeAttributesManagerImpl#writeLock and may have some influence in a large > cluster. We have found nodes UI of a large cluster is opened slowly and most > time it's waiting for the lock in NodeAttributesManagerImpl. I think this > updating should be called only when necessary to enhance the performance of > related process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717395#comment-16717395
]
Akhil PB edited comment on YARN-9089 at 12/11/18 3:45 PM:
--
[~eyang] Yes, It is the convention we are following to use initializers in
ember (here loader.js) to set ENV configs. As you go through loader.js, you can
see that we are setting all the env configs in loader.js, which works in both
dev mode (running ui2 in local development) and embedded mode (running UI2 from
RM - http//rm-host:8088/ui2).
For example, please follow the steps of how cluster-id is being set to the ENV.
Please refere {{initializers/loader.js}} for more clarity.
{code}
var clusterIdFromYARN = getClusterIdFromYARN(rmhost, application);
ENV.clusterId = clusterIdFromYARN;
{code}
{code}
function getClusterIdFromYARN(rmhost, application) {
var httpUrl = window.location.protocol + '//' +
(ENV.hosts.localBaseAddress? ENV.hosts.localBaseAddress + '/' : '') +
rmhost;
httpUrl += '/conf?name=yarn.resourcemanager.cluster-id';
Ember.Logger.log("Get cluster-id URL is: " + httpUrl);
var clusterId = "";
$.ajax({
type: 'GET',
dataType: 'json',
async: false,
context: this,
url: httpUrl,
success: function(data) {
clusterId = data.property.value;
Ember.Logger.log("Cluster Id from RM: " + clusterId);
application.advanceReadiness();
},
error: function() {
application.advanceReadiness();
}
});
return clusterId;
}
{code}
Then we can access the property from env (here clusterId) as below from any
adapters or controllers, since we are injecting env into controller, adapters
and routes.
{code}
var clusterId = this.get("env.app.clusterId");
{code}
was (Author: akhilpb):
[~eyang] Yes, It is the convention we are following to use initializers in
ember (here loader.js) to set ENV configs. As you go through loader.js, you can
see that we are setting all the env configs in loader.js, which works in both
dev mode (running ui2 in local development) and embedded mode (running UI2 from
RM - http//rm-host:8088/ui2).
For example, please follow the steps of how cluster-id is being set to the ENV.
Please refere {{initializers/loader.js}} for more clarity.
{code}
var clusterIdFromYARN = getClusterIdFromYARN(rmhost, application);
ENV.clusterId = clusterIdFromYARN;
{code}
{code}
function getClusterIdFromYARN(rmhost, application) {
var httpUrl = window.location.protocol + '//' +
(ENV.hosts.localBaseAddress? ENV.hosts.localBaseAddress + '/' : '') +
rmhost;
httpUrl += '/conf?name=yarn.resourcemanager.cluster-id';
Ember.Logger.log("Get cluster-id URL is: " + httpUrl);
var clusterId = "";
$.ajax({
type: 'GET',
dataType: 'json',
async: false,
context: this,
url: httpUrl,
success: function(data) {
clusterId = data.property.value;
Ember.Logger.log("Cluster Id from RM: " + clusterId);
application.advanceReadiness();
},
error: function() {
application.advanceReadiness();
}
});
return clusterId;
}
{code}
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9089) Add Terminal Link to Service component instance page for UI2
[
https://issues.apache.org/jira/browse/YARN-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717395#comment-16717395
]
Akhil PB commented on YARN-9089:
[~eyang] Yes, It is the convention we are following to use initializers in
ember (here loader.js) to set ENV configs. As you go through loader.js, you can
see that we are setting all the env configs in loader.js, which works in both
dev mode (running ui2 in local development) and embedded mode (running UI2 from
RM - http//rm-host:8088/ui2).
For example, please follow the steps of how cluster-id is being set to the ENV.
Please refere {{initializers/loader.js}} for more clarity.
{code}
var clusterIdFromYARN = getClusterIdFromYARN(rmhost, application);
ENV.clusterId = clusterIdFromYARN;
{code}
{code}
function getClusterIdFromYARN(rmhost, application) {
var httpUrl = window.location.protocol + '//' +
(ENV.hosts.localBaseAddress? ENV.hosts.localBaseAddress + '/' : '') +
rmhost;
httpUrl += '/conf?name=yarn.resourcemanager.cluster-id';
Ember.Logger.log("Get cluster-id URL is: " + httpUrl);
var clusterId = "";
$.ajax({
type: 'GET',
dataType: 'json',
async: false,
context: this,
url: httpUrl,
success: function(data) {
clusterId = data.property.value;
Ember.Logger.log("Cluster Id from RM: " + clusterId);
application.advanceReadiness();
},
error: function() {
application.advanceReadiness();
}
});
return clusterId;
}
{code}
> Add Terminal Link to Service component instance page for UI2
>
>
> Key: YARN-9089
> URL: https://issues.apache.org/jira/browse/YARN-9089
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9089.001.patch
>
>
> In UI2, Service > Component > Component Instance uses Timeline server to
> aggregate information about Service component instance. Timeline server does
> not have the full information like the port number of the node manager, or
> the web protocol used by the node manager. It requires some changes to
> aggregate node manager information into Timeline server in order to compute
> the Terminal link. For reducing the scope of YARN-8914, it is better file
> this as a separate task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
[
https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717317#comment-16717317
]
Hadoop QA commented on YARN-8925:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 15s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 344 unchanged - 0 fixed = 350 total (was 344) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
36s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 15s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}209m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests
[jira] [Commented] (YARN-9047) FairScheduler: default resource calculator is not resource type aware
[
https://issues.apache.org/jira/browse/YARN-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717238#comment-16717238
]
Hadoop QA commented on YARN-9047:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 16s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 32 new + 227 unchanged - 0 fixed = 259 total (was 227) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 41s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9047 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951343/YARN-9047.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b9dbbf79f680 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ff8580 |
| maven | v
[jira] [Updated] (YARN-7291) Better input parsing for resource in allocation file
[ https://issues.apache.org/jira/browse/YARN-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-7291: - Attachment: YARN-7291.002.patch > Better input parsing for resource in allocation file > > > Key: YARN-7291 > URL: https://issues.apache.org/jira/browse/YARN-7291 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Szilard Nemeth >Priority: Minor > Labels: newbie > Attachments: YARN-7291.001.patch, YARN-7291.002.patch > > > When you set max/min share for queues in fair scheduler allocation file, > "1024 mb, 2 4 vcores" is parsed the same as "1024 mb, 4 vcores" without any > issue, the same to "50% memory, 50% 100%cpu" which is parsed the same as "50% > memory, 100%cpu". That causes confusing. We should fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with file localization feature
[
https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717119#comment-16717119
]
Hadoop QA commented on YARN-9008:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 1 new + 205 unchanged - 0 fixed = 206 total (was 205)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 5s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9008 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951350/YARN-9008-006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5452562c4d35 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ff8580 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22835/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22835/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
|
[jira] [Updated] (YARN-9008) Extend YARN distributed shell with file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9008: --- Attachment: YARN-9008-006.patch > Extend YARN distributed shell with file localization feature > > > Key: YARN-9008 > URL: https://issues.apache.org/jira/browse/YARN-9008 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9008-001.patch, YARN-9008-002.patch, > YARN-9008-003.patch, YARN-9008-004.patch, YARN-9008-005.patch, > YARN-9008-006.patch > > > YARN distributed shell is a very handy tool to test various features of YARN. > However, it lacks support for file localization - that is, you define files > in the command line that you wish to be localized remotely. This can be > extremely useful in certain scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9108) FederationIntercepter merge home and second response local variable spell mistake
[
https://issues.apache.org/jira/browse/YARN-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Morty Zhong updated YARN-9108:
--
Description:
method 'mergeAllocateResponse' in class FederationIntercepter.java line 1315
the left variable `par2` should be `par1`
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par2.getContainers().addAll(par2.getContainers());
}
{code}
should be
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par1.getContainers().addAll(par2.getContainers());//edited line
}
{code}
was:
method 'mergeAllocateResponse' in class FederationIntercepter.java line 1315
the left variable `par2` should be `par1`
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par2.getContainers().addAll(par2.getContainers());
}
{code}
should be
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par1.getContainers().addAll(par2.getContainers());
}
{code}
> FederationIntercepter merge home and second response local variable spell
> mistake
> -
>
> Key: YARN-9108
> URL: https://issues.apache.org/jira/browse/YARN-9108
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.3.0
>Reporter: Morty Zhong
>Priority: Minor
>
> method 'mergeAllocateResponse' in class FederationIntercepter.java line 1315
> the left variable `par2` should be `par1`
> {code:java}
> if (par1 != null && par2 != null) {
> par1.getResourceRequest().addAll(par2.getResourceRequest());
> par2.getContainers().addAll(par2.getContainers());
> }
> {code}
> should be
> {code:java}
> if (par1 != null && par2 != null) {
> par1.getResourceRequest().addAll(par2.getResourceRequest());
> par1.getContainers().addAll(par2.getContainers());//edited line
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9108) FederationIntercepter merge home and second response local variable spell mistake
Morty Zhong created YARN-9108:
-
Summary: FederationIntercepter merge home and second response
local variable spell mistake
Key: YARN-9108
URL: https://issues.apache.org/jira/browse/YARN-9108
Project: Hadoop YARN
Issue Type: Bug
Components: federation
Affects Versions: 3.3.0
Reporter: Morty Zhong
method 'mergeAllocateResponse' in class FederationIntercepter.java line 1315
the left variable `par2` should be `par1`
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par2.getContainers().addAll(par2.getContainers());
}
{code}
should be
{code:java}
if (par1 != null && par2 != null) {
par1.getResourceRequest().addAll(par2.getResourceRequest());
par1.getContainers().addAll(par2.getContainers());
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8489) Need to support "dominant" component concept inside YARN service
[ https://issues.apache.org/jira/browse/YARN-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16716694#comment-16716694 ] Zac Zhou commented on YARN-8489: @[~suma.shivaprasad] any Updates? Or would you mind if I take it, as this jira blocks terminating submarine job gracefully. > Need to support "dominant" component concept inside YARN service > > > Key: YARN-8489 > URL: https://issues.apache.org/jira/browse/YARN-8489 > Project: Hadoop YARN > Issue Type: Task > Components: yarn-native-services >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Major > > Existing YARN service support termination policy for different restart > policies. For example ALWAYS means service will not be terminated. And NEVER > means if all component terminated, service will be terminated. > The name "dominant" might not be most appropriate , we can figure out better > names. But in simple, it means, a dominant component which final state will > determine job's final state regardless of other components. > Use cases: > 1) Tensorflow job has master/worker/services/tensorboard. Once master goes to > final state, no matter if it is succeeded or failed, we should terminate > ps/tensorboard/workers. And the mark the job to succeeded/failed. > 2) Not sure if it is a real-world use case: A service which has multiple > component, some component is not restartable. For such services, if a > component is failed, we should mark the whole service to failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8489) Need to support "dominant" component concept inside YARN service
[ https://issues.apache.org/jira/browse/YARN-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16716694#comment-16716694 ] Zac Zhou edited comment on YARN-8489 at 12/11/18 9:47 AM: -- [~suma.shivaprasad] any Updates? Or would you mind if I take it, as this jira blocks terminating submarine job gracefully. was (Author: yuan_zac): @[~suma.shivaprasad] any Updates? Or would you mind if I take it, as this jira blocks terminating submarine job gracefully. > Need to support "dominant" component concept inside YARN service > > > Key: YARN-8489 > URL: https://issues.apache.org/jira/browse/YARN-8489 > Project: Hadoop YARN > Issue Type: Task > Components: yarn-native-services >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Major > > Existing YARN service support termination policy for different restart > policies. For example ALWAYS means service will not be terminated. And NEVER > means if all component terminated, service will be terminated. > The name "dominant" might not be most appropriate , we can figure out better > names. But in simple, it means, a dominant component which final state will > determine job's final state regardless of other components. > Use cases: > 1) Tensorflow job has master/worker/services/tensorboard. Once master goes to > final state, no matter if it is succeeded or failed, we should terminate > ps/tensorboard/workers. And the mark the job to succeeded/failed. > 2) Not sure if it is a real-world use case: A service which has multiple > component, some component is not restartable. For such services, if a > component is failed, we should mark the whole service to failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9047) FairScheduler: default resource calculator is not resource type aware
[
https://issues.apache.org/jira/browse/YARN-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1671#comment-1671
]
Szilard Nemeth commented on YARN-9047:
--
[~wilfreds]: As discussed offline, we need to use the
DominantResourceCalculator in all cases.
I ran all the tests locally (not just FS but all YARN tests), they all passed
so I expect the same from Jenkins.
So in this sense, this is ready for review.
> FairScheduler: default resource calculator is not resource type aware
> -
>
> Key: YARN-9047
> URL: https://issues.apache.org/jira/browse/YARN-9047
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Wilfred Spiegelenburg
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9047.001.patch, YARN-9047.002.patch,
> YARN-9047.003.patch
>
>
> The FairScheduler#getResourceCalculator always returns the default resource
> calculator. The default calculator is not resource type aware and should only
> be used if there are no resource types configured.
> We need to make sure that in we the direct hard code reference to
> {{RESOURCE_CALCULATOR}} is either safe to use in all cases or is not used in
> the scheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9047) FairScheduler: default resource calculator is not resource type aware
[
https://issues.apache.org/jira/browse/YARN-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-9047:
-
Attachment: YARN-9047.003.patch
> FairScheduler: default resource calculator is not resource type aware
> -
>
> Key: YARN-9047
> URL: https://issues.apache.org/jira/browse/YARN-9047
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Wilfred Spiegelenburg
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9047.001.patch, YARN-9047.002.patch,
> YARN-9047.003.patch
>
>
> The FairScheduler#getResourceCalculator always returns the default resource
> calculator. The default calculator is not resource type aware and should only
> be used if there are no resource types configured.
> We need to make sure that in we the direct hard code reference to
> {{RESOURCE_CALCULATOR}} is either safe to use in all cases or is not used in
> the scheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
[ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16716634#comment-16716634 ] Tao Yang commented on YARN-8925: Hi, [~cheersyang] Just noticed DISTRIBUTED and CENTRALIZED attributes can be exist together. I wrongly thought node attributes should choose DISTRIBUTED or CENTRALIZED like node label. Attached v10 patch to filter DISTRIBUTED attributes before comparing current and new attributes in ResourceTrackerService#nodeHeartbeat and add UT for this scenario in TestResourceTrackerService#testNodeHeartbeatWithInvalidNodeAttributes. Could you please help to review again. Thanks. > Updating distributed node attributes only when necessary > > > Key: YARN-8925 > URL: https://issues.apache.org/jira/browse/YARN-8925 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: performance > Attachments: YARN-8925.001.patch, YARN-8925.002.patch, > YARN-8925.003.patch, YARN-8925.004.patch, YARN-8925.005.patch, > YARN-8925.006.patch, YARN-8925.007.patch, YARN-8925.008.patch, > YARN-8925.009.patch, YARN-8925.010.patch > > > Currently if distributed node attributes exist, even though there is no > change, updating for distributed node attributes will happen in every > heartbeat between NM and RM. Updating process will hold > NodeAttributesManagerImpl#writeLock and may have some influence in a large > cluster. We have found nodes UI of a large cluster is opened slowly and most > time it's waiting for the lock in NodeAttributesManagerImpl. I think this > updating should be called only when necessary to enhance the performance of > related process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8925) Updating distributed node attributes only when necessary
[ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-8925: --- Attachment: YARN-8925.010.patch > Updating distributed node attributes only when necessary > > > Key: YARN-8925 > URL: https://issues.apache.org/jira/browse/YARN-8925 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: performance > Attachments: YARN-8925.001.patch, YARN-8925.002.patch, > YARN-8925.003.patch, YARN-8925.004.patch, YARN-8925.005.patch, > YARN-8925.006.patch, YARN-8925.007.patch, YARN-8925.008.patch, > YARN-8925.009.patch, YARN-8925.010.patch > > > Currently if distributed node attributes exist, even though there is no > change, updating for distributed node attributes will happen in every > heartbeat between NM and RM. Updating process will hold > NodeAttributesManagerImpl#writeLock and may have some influence in a large > cluster. We have found nodes UI of a large cluster is opened slowly and most > time it's waiting for the lock in NodeAttributesManagerImpl. I think this > updating should be called only when necessary to enhance the performance of > related process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
