[jira] [Commented] (YARN-10494) CLI tool for docker-to-squashfs conversion (pure Java)
[ https://issues.apache.org/jira/browse/YARN-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243597#comment-17243597 ] Eric Badger commented on YARN-10494: I'm pretty much in line with [~ccondit]'s opinion here. This squashfs code definitely isn't specific to Hadoop and could be a separate project. However, that's a lot of work and I'm not sure there's any real benefit there. Also, I think a PR is much better for a large commit because it becomes much easier to make comments in line with the code instead of having to explain where the issues are in a JIRA comment > CLI tool for docker-to-squashfs conversion (pure Java) > -- > > Key: YARN-10494 > URL: https://issues.apache.org/jira/browse/YARN-10494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Major > Labels: pull-request-available > Attachments: YARN-10494.001.patch, > docker-to-squashfs-conversion-tool-design.pdf > > Time Spent: 40m > Remaining Estimate: 0h > > *YARN-9564* defines a docker-to-squashfs image conversion tool that relies on > python2, multiple libraries, squashfs-tools and root access in order to > convert Docker images to squashfs images for use with the runc container > runtime in YARN. > *YARN-9943* was created to investigate alternatives, as the response to > merging YARN-9564 has not been very positive. This proposal outlines the > design for a CLI conversion tool in 100% pure Java that will work out of the > box. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10516) In HA mode, when one Resource Manager has networking issue, getTokenService() should not throw runtime exception
[
https://issues.apache.org/jira/browse/YARN-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated YARN-10516:
---
Description:
We have observed one issue from YARN client around this piece of code:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
While
{code:java}
services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
defaultAddr, defaultPort)) .toString());
{code}
is being called, buildTokenService() fails and will throw runtime
exception, more specifically, UnknownHostException from here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
*while one of the RM host was having networking issue* that IP cannot be
resolved.
This runtime exception then floats all the way up to our application and causes
MR job submission failed.
In my opinion, since we have HA here, multiple RMs are still alive and
available. We should catch this exception in getTokenService() and handle it
properly, instead of failing the whole action.
Would like to hear your opinion on this, if agreed, I will provide a patch on
this. Thank you.
was:
We have observed one issue from YARN client around this piece of code:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
While
{code:java}
services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
defaultAddr, defaultPort)) .toString());
{code}
is being called, "yarnConf.getSocketAddr" will throw runtime exception, more
specifically, UnknownHostException from here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
while one of the RM host was having networking issue that IP cannot be
resolved.
This runtime exception then floats all the way up to our application and causes
MR job submission failed.
In my opinion, since we have HA here, multiple RMs are still alive and
available. We should catch this exception in getTokenService() and handle it
properly, instead of failing the whole action.
Would like to hear your opinion on this, if agreed, I will provide a patch on
this. Thank you.
> In HA mode, when one Resource Manager has networking issue, getTokenService()
> should not throw runtime exception
>
>
> Key: YARN-10516
> URL: https://issues.apache.org/jira/browse/YARN-10516
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Xu Cang
>Priority: Minor
>
> We have observed one issue from YARN client around this piece of code:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
>
> While
> {code:java}
> services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
> defaultAddr, defaultPort)) .toString());
>
> {code}
> is being called, buildTokenService() fails and will throw runtime
> exception, more specifically, UnknownHostException from here:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
> *while one of the RM host was having networking issue* that IP cannot be
> resolved.
> This runtime exception then floats all the way up to our application and
> causes MR job submission failed.
> In my opinion, since we have HA here, multiple RMs are still alive and
> available. We should catch this exception in getTokenService() and handle it
> properly, instead of failing the whole action.
>
>
> Would like to hear your opinion on this, if agreed, I will provide a patch on
> this. Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10516) In HA mode, when one Resource Manager has networking issue, getTokenService() should not throw runtime exception
[
https://issues.apache.org/jira/browse/YARN-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated YARN-10516:
---
Description:
We have observed one issue from YARN client around this piece of code:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
While
{code:java}
services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
defaultAddr, defaultPort)) .toString());
{code}
is being called, "yarnConf.getSocketAddr" will throw runtime exception, more
specifically, UnknownHostException from here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
while one of the RM host was having networking issue that IP cannot be
resolved.
This runtime exception then floats all the way up to our application and causes
MR job submission failed.
In my opinion, since we have HA here, multiple RMs are still alive and
available. We should catch this exception in getTokenService() and handle it
properly, instead of failing the whole action.
Would like to hear your opinion on this, if agreed, I will provide a patch on
this. Thank you.
was:
We have observed one issue from YARN client around this piece of code:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
While
{code:java}
services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
defaultAddr, defaultPort)) .toString());
{code}
Is being called, "yarnConf.getSocketAddr" will throw runtime exception, more
specifically, UnknownHostException from here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
while one of the RM host was having networking issue that IP cannot be resolved.
This runtime exception then floats all the way into our application and cause
MR job submission failed.
In my opinion, since we have HA here, multiple RMs are still alive and
available. We should catch this exception in getTokenService() and handle it
properly.
Would like to hear your opinion on this, if agreed, I will provide a patch on
this. Thank you.
> In HA mode, when one Resource Manager has networking issue, getTokenService()
> should not throw runtime exception
>
>
> Key: YARN-10516
> URL: https://issues.apache.org/jira/browse/YARN-10516
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Xu Cang
>Priority: Minor
>
> We have observed one issue from YARN client around this piece of code:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
>
> While
> {code:java}
> services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
> defaultAddr, defaultPort)) .toString());
>
> {code}
> is being called, "yarnConf.getSocketAddr" will throw runtime exception,
> more specifically, UnknownHostException from here:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
> while one of the RM host was having networking issue that IP cannot be
> resolved.
> This runtime exception then floats all the way up to our application and
> causes MR job submission failed.
> In my opinion, since we have HA here, multiple RMs are still alive and
> available. We should catch this exception in getTokenService() and handle it
> properly, instead of failing the whole action.
>
>
> Would like to hear your opinion on this, if agreed, I will provide a patch on
> this. Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10516) In HA mode, when one Resource Manager has networking issue, getTokenService() should not throw runtime exception
Xu Cang created YARN-10516:
--
Summary: In HA mode, when one Resource Manager has networking
issue, getTokenService() should not throw runtime exception
Key: YARN-10516
URL: https://issues.apache.org/jira/browse/YARN-10516
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Xu Cang
We have observed one issue from YARN client around this piece of code:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
While
{code:java}
services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address,
defaultAddr, defaultPort)) .toString());
{code}
Is being called, "yarnConf.getSocketAddr" will throw runtime exception, more
specifically, UnknownHostException from here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
while one of the RM host was having networking issue that IP cannot be resolved.
This runtime exception then floats all the way into our application and cause
MR job submission failed.
In my opinion, since we have HA here, multiple RMs are still alive and
available. We should catch this exception in getTokenService() and handle it
properly.
Would like to hear your opinion on this, if agreed, I will provide a patch on
this. Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10494) CLI tool for docker-to-squashfs conversion (pure Java)
[ https://issues.apache.org/jira/browse/YARN-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243493#comment-17243493 ] Craig Condit commented on YARN-10494: - [~Jim_Brennan], I don't disagree that the squashfs code could be its own project, however that does seem like quite a bit to bite off. I'm also not sure how large of audience outside Hadoop there would be, given squashfs iwas originally designed to support Linux initial ramdisk functionality. Would you be ok with the idea of adding this within Hadoop for now, with the idea to split it out if and when there's demand? > CLI tool for docker-to-squashfs conversion (pure Java) > -- > > Key: YARN-10494 > URL: https://issues.apache.org/jira/browse/YARN-10494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Major > Labels: pull-request-available > Attachments: YARN-10494.001.patch, > docker-to-squashfs-conversion-tool-design.pdf > > Time Spent: 20m > Remaining Estimate: 0h > > *YARN-9564* defines a docker-to-squashfs image conversion tool that relies on > python2, multiple libraries, squashfs-tools and root access in order to > convert Docker images to squashfs images for use with the runc container > runtime in YARN. > *YARN-9943* was created to investigate alternatives, as the response to > merging YARN-9564 has not been very positive. This proposal outlines the > design for a CLI conversion tool in 100% pure Java that will work out of the > box. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10515:
Component/s: test
> Fix flaky test
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> --
>
> Key: YARN-10515
> URL: https://issues.apache.org/jira/browse/YARN-10515
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10515-001.patch
>
>
> The testcase
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> sometimes fails with the following error:
> {noformat}
> org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:884)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1018)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:134)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation$5.(TestCapacitySchedulerAutoQueueCreation.java:873)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags(TestCapacitySchedulerAutoQueueCreation.java:873)
> ...
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> PartitionQueueMetrics,partition=,q0=root,q1=a already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionQueueMetrics(QueueMetrics.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.setAvailableResourcesToQueue(QueueMetrics.java:513)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateQueueStatistics(CSQueueUtils.java:308)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:412)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:350)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:119)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:289)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:752)
> {noformat}
> We have to reset queue metrics before running this test to make sure it
> passes.
--
[jira] [Updated] (YARN-10484) The number of containers of a single node in the heatmap was not displayed correctly
[ https://issues.apache.org/jira/browse/YARN-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-10484: -- Labels: pull-request-available (was: ) > The number of containers of a single node in the heatmap was not displayed > correctly > > > Key: YARN-10484 > URL: https://issues.apache.org/jira/browse/YARN-10484 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1 >Reporter: akiyamaneko >Priority: Major > Labels: pull-request-available > Attachments: screenshot-1.png, screenshot-2.png > > Time Spent: 10m > Remaining Estimate: 0h > > there is a problem with the logic of calculating the number of *total > Containers* under the same rack, which causes different nodes in the same > rack to display totally different *totalContainers* values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243393#comment-17243393
]
Hadoop QA commented on YARN-10515:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
49s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 6s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 26s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} the
[jira] [Resolved] (YARN-10511) Update yarn.nodemanager.env-whitelist value in docs
[
https://issues.apache.org/jira/browse/YARN-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-10511.
--
Fix Version/s: 3.2.3
3.1.5
3.3.1
3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Merged the PR into trunk, branch-3.3, branch-3.2, and branch-3.1. Thank you
[~ascarpino] for your contribution!
> Update yarn.nodemanager.env-whitelist value in docs
> ---
>
> Key: YARN-10511
> URL: https://issues.apache.org/jira/browse/YARN-10511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Andrea Scarpino
>Assignee: Andrea Scarpino
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Hadoop Cluster
> Setup|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html]
> documentation page references an old value for the
> {{yarn.nodemanager.env-whitelist}} variable because the value referenced in
> that page has been changed since YARN-6694.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10511) Update yarn.nodemanager.env-whitelist value in docs
[
https://issues.apache.org/jira/browse/YARN-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10511:
Assignee: Andrea Scarpino
Summary: Update yarn.nodemanager.env-whitelist value in docs (was: Update
yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup)
> Update yarn.nodemanager.env-whitelist value in docs
> ---
>
> Key: YARN-10511
> URL: https://issues.apache.org/jira/browse/YARN-10511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Andrea Scarpino
>Assignee: Andrea Scarpino
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Hadoop Cluster
> Setup|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html]
> documentation page references an old value for the
> {{yarn.nodemanager.env-whitelist}} variable because the value referenced in
> that page has been changed since YARN-6694.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10511) Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
[
https://issues.apache.org/jira/browse/YARN-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10511:
-
Labels: pull-request-available (was: documentation pull-request-available)
> Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
> ---
>
> Key: YARN-10511
> URL: https://issues.apache.org/jira/browse/YARN-10511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Andrea Scarpino
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Hadoop Cluster
> Setup|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html]
> documentation page references an old value for the
> {{yarn.nodemanager.env-whitelist}} variable because the value referenced in
> that page has been changed since YARN-6694.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10511) Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
[
https://issues.apache.org/jira/browse/YARN-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10511:
-
Component/s: documentation
Issue Type: Bug (was: Improvement)
Priority: Minor (was: Trivial)
> Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
> ---
>
> Key: YARN-10511
> URL: https://issues.apache.org/jira/browse/YARN-10511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Andrea Scarpino
>Priority: Minor
> Labels: documentation, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The [Hadoop Cluster
> Setup|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html]
> documentation page references an old value for the
> {{yarn.nodemanager.env-whitelist}} variable because the value referenced in
> that page has been changed since YARN-6694.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10494) CLI tool for docker-to-squashfs conversion (pure Java)
[ https://issues.apache.org/jira/browse/YARN-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243242#comment-17243242 ] Jim Brennan commented on YARN-10494: I'm not sure that a PR is better than a patch for something this big. One initial comment though, there is a lot of code here which is essentially squashf-tools-for-java. I wonder if that portion should be a separate open source project? > CLI tool for docker-to-squashfs conversion (pure Java) > -- > > Key: YARN-10494 > URL: https://issues.apache.org/jira/browse/YARN-10494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Major > Labels: pull-request-available > Attachments: YARN-10494.001.patch, > docker-to-squashfs-conversion-tool-design.pdf > > Time Spent: 20m > Remaining Estimate: 0h > > *YARN-9564* defines a docker-to-squashfs image conversion tool that relies on > python2, multiple libraries, squashfs-tools and root access in order to > convert Docker images to squashfs images for use with the runc container > runtime in YARN. > *YARN-9943* was created to investigate alternatives, as the response to > merging YARN-9564 has not been very positive. This proposal outlines the > design for a CLI conversion tool in 100% pure Java that will work out of the > box. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10514) Introduce a dominant resource based schedule policy to increase the resource utilization, avoid heavy cluster resource fragments.
[
https://issues.apache.org/jira/browse/YARN-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243188#comment-17243188
]
Hadoop QA commented on YARN-10514:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
3s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
10s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 23s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
14s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
38s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/358/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 38s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/358/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
5s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
5s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 26s{color} |
[jira] [Commented] (YARN-10031) Create a general purpose log request with additional query parameters
[
https://issues.apache.org/jira/browse/YARN-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243160#comment-17243160
]
Hadoop QA commented on YARN-10031:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
26s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 58s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
54s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
20s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
24s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 42s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/356/artifact/out/diff-checkstyle-root.txt{color}
| {color:orange} root: The patch generated 8 new + 39 unchanged - 0 fixed = 47
total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243155#comment-17243155
]
Hadoop QA commented on YARN-10515:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 36s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
2s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
15s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/359/artifact/out/patch-shadedclient.txt{color}
| {color:red} patch has errors when building and testing our client artifacts.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} |
[jira] [Commented] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243149#comment-17243149 ] Peter Bacsko commented on YARN-10496: - _"From the design doc, one proposal is to define max capacity for weighted queues in terms of percentage of the cluster rather than percentage of the immediate parent. I would oppose this since max capacity in CS has always been in relative to the immediate parent."_ This is a valid concern. Unfortunately for us, in Fair Scheduler, the percentages are relative to the overall cluster capacity. We were not entirely sure about this when we put together the material, but I examined the FS part a bit deeply and these classes are the main point of interest: * org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.allocation.AllocationFileQueueParser.loadQueue() - [https://github.com/apache/hadoop/blob/fd6be5898ad1a650e3bceacb8169a53520da57e5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L141-L145] * org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.ConfigurableResource - [https://github.com/apache/hadoop/blob/5cc7873a4723a6c8e8e001d008fcd522eec0433d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ConfigurableResource.java#L44-L47] * ConfigurableResource.getResource() - [https://github.com/apache/hadoop/blob/5cc7873a4723a6c8e8e001d008fcd522eec0433d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ConfigurableResource.java#L79-L95] So when we calculate the maximum resource for a queue, it's expressed in the percentage of the overall cluster capacity. I think we can do multiple things: * We just ignore this and go the CS-way, meaning that the calculation will be based on the parent. This will be inconvenient for legacy FS users. * Add a feature flag to indicate which calculation you want. * Add a new format to "maximum-capacity". Like "c:50%" or "cluster:[50%]" to indicate what the percentage refers to. Thoughts? > [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler > - > > Key: YARN-10496 > URL: https://issues.apache.org/jira/browse/YARN-10496 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacity scheduler >Reporter: Wangda Tan >Priority: Major > > CapacityScheduler today doesn’t support an auto queue creation which is > flexible enough. The current constraints: > * Only leaf queues can be auto-created > * A parent can only have either static queues or dynamic ones. This causes > multiple constraints. For example: > * It isn’t possible to have a VIP user like Alice with a static queue > root.user.alice with 50% capacity while the other user queues (under > root.user) are created dynamically and they share the remaining 50% of > resources. > > * In comparison, FairScheduler allows the following scenarios, Capacity > Scheduler doesn’t: > ** This implies that there is no possibility to have both dynamically > created and static queues at the same time under root > * A new queue needs to be created under an existing parent, while the parent > already has static queues > * Nested queue mapping policy, like in the following example: > | > > | > * Here two levels of queues may need to be created > If an application belongs to user _alice_ (who has the primary_group of > _engineering_), the scheduler checks whether _root.engineering_ exists, if it > doesn’t, it’ll be created. Then scheduler checks whether > _root.engineering.alice_ exists, and creates it if it doesn't. > > When we try to move users from FairScheduler to CapacityScheduler, we face > feature gaps which blocks users migrate from FS to CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10515:
Attachment: YARN-10515-001.patch
> Fix flaky test
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> --
>
> Key: YARN-10515
> URL: https://issues.apache.org/jira/browse/YARN-10515
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10515-001.patch
>
>
> The testcase
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> sometimes fails with the following error:
> {noformat}
> org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:884)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1018)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:134)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation$5.(TestCapacitySchedulerAutoQueueCreation.java:873)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags(TestCapacitySchedulerAutoQueueCreation.java:873)
> ...
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> PartitionQueueMetrics,partition=,q0=root,q1=a already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionQueueMetrics(QueueMetrics.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.setAvailableResourcesToQueue(QueueMetrics.java:513)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateQueueStatistics(CSQueueUtils.java:308)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:412)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:350)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:119)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:289)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:752)
> {noformat}
> We have to reset queue metrics before running this test to make sure it
> passes.
--
This message
[jira] [Updated] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10515:
Description:
The testcase
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
sometimes fails with the following error:
{noformat}
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:884)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1018)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:158)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:134)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:130)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation$5.(TestCapacitySchedulerAutoQueueCreation.java:873)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags(TestCapacitySchedulerAutoQueueCreation.java:873)
...
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
PartitionQueueMetrics,partition=,q0=root,q1=a already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionQueueMetrics(QueueMetrics.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.setAvailableResourcesToQueue(QueueMetrics.java:513)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateQueueStatistics(CSQueueUtils.java:308)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:412)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:350)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:137)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:119)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:52)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:57)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:261)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:289)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:162)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:752)
{noformat}
We have to reset queue metrics before running this test to make sure it passes.
was:
The testcase
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
sometimes fails with the following error:
{noformat}
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
at
[jira] [Commented] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243113#comment-17243113
]
Hadoop QA commented on YARN-7200:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 56s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
|
[jira] [Created] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
Peter Bacsko created YARN-10515:
---
Summary: Fix flaky test
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
Key: YARN-10515
URL: https://issues.apache.org/jira/browse/YARN-10515
Project: Hadoop YARN
Issue Type: Bug
Reporter: Peter Bacsko
Assignee: Peter Bacsko
The testcase
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
sometimes fails with the following error:
{noformat}
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:884)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1018)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:158)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:134)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:130)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation$5.(TestCapacitySchedulerAutoQueueCreation.java:873)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags(TestCapacitySchedulerAutoQueueCreation.java:873)
{noformat}
We have to reset queue metrics before running this test to make sure it passes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10507) Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml
[
https://issues.apache.org/jira/browse/YARN-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243080#comment-17243080
]
Peter Bacsko commented on YARN-10507:
-
Failed tests are unrelated but I know how to fix
TestCapacitySchedulerAutoQueueCreation.
> Add the capability to fs2cs to write the converted placement rules inside
> capacity-scheduler.xml
>
>
> Key: YARN-10507
> URL: https://issues.apache.org/jira/browse/YARN-10507
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10507-001.patch, YARN-10507-002.patch,
> YARN-10507-003.patch, YARN-10507-004.patch, YARN-10507-005.patch,
> YARN-10507-006.patch
>
>
> Currently, fs2cs tool generates a separate {{mapping-rules.json}} file when
> it converts the placement rules.
> However, we also support having the JSON inlined inside
> {{capacity-scheduler.xml}}. Add a command line switch so that we can choose
> the desired output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10507) Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml
[
https://issues.apache.org/jira/browse/YARN-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243075#comment-17243075
]
Peter Bacsko commented on YARN-10507:
-
[~snemeth] could you review this?
There might be parts which needs to be explained.
> Add the capability to fs2cs to write the converted placement rules inside
> capacity-scheduler.xml
>
>
> Key: YARN-10507
> URL: https://issues.apache.org/jira/browse/YARN-10507
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10507-001.patch, YARN-10507-002.patch,
> YARN-10507-003.patch, YARN-10507-004.patch, YARN-10507-005.patch,
> YARN-10507-006.patch
>
>
> Currently, fs2cs tool generates a separate {{mapping-rules.json}} file when
> it converts the placement rules.
> However, we also support having the JSON inlined inside
> {{capacity-scheduler.xml}}. Add a command line switch so that we can choose
> the desired output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10514) Introduce a dominant resource based schedule policy to increase the resource utilization, avoid heavy cluster resource fragments.
[ https://issues.apache.org/jira/browse/YARN-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243071#comment-17243071 ] zhuqi edited comment on YARN-10514 at 12/3/20, 10:17 AM: - [~leftnoteasy] [~tangzhankun] [~prabhujoseph] [~sunil.gov...@gmail.com] [~jiwq] If you any advice about this proposal. I submit a draft patch to support async multi node scheduling mode in CS, but i think in heartbeat schedule mode in CS/FS, we also should to handle resource resource fragments, to help get better resource utilization. was (Author: zhuqi): [~leftnoteasy] [~tangzhankun] [~prabhujoseph] [~sunil.gov...@gmail.com] [~jiwq] If you any advice about this proposal. I submit a draft patch to support async multi node scheduling mode, but i think in heartbeat schedule mode we also should to handle resource resource fragments, to help get better resource utilization. > Introduce a dominant resource based schedule policy to increase the resource > utilization, avoid heavy cluster resource fragments. > - > > Key: YARN-10514 > URL: https://issues.apache.org/jira/browse/YARN-10514 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0, 3.4.0 >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10514.001.patch > > > When we schedule in multi node lookup policy for async scheduling, or just > use heartbeat update based scheduling, we both meet scheduling fragments. > When cpu-intensive jobs or gpu-intensive or memory-intensive etc, the cluster > will meet heavy waste of resources, so this issue will help to move scheduler > support dominant resource based schedule, to help our cluster get better > resource utilization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Auto Queue Creation Check RM REST API impact
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10512: - Summary: CS Auto Queue Creation Check RM REST API impact (was: CS Auto Queue Creation Check RM rest API impact) > CS Auto Queue Creation Check RM REST API impact > --- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10513: - Summary: CS Flexible Auto Queue Creation RM UIv2 modifications (was: CS Auto Queue Creation RM UIv2 modifications) > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation Check RM REST API impact
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10512: - Summary: CS Flexible Auto Queue Creation Check RM REST API impact (was: CS Auto Queue Creation Check RM REST API impact) > CS Flexible Auto Queue Creation Check RM REST API impact > > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10491) Fix deprecation warnings in SLSWebApp.java
[
https://issues.apache.org/jira/browse/YARN-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Kumar reassigned YARN-10491:
--
Assignee: Ankit Kumar
> Fix deprecation warnings in SLSWebApp.java
> --
>
> Key: YARN-10491
> URL: https://issues.apache.org/jira/browse/YARN-10491
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Ankit Kumar
>Priority: Minor
> Labels: newbie
>
> {code}
> simulateInfoTemplate = IOUtils.toString(
> cl.getResourceAsStream("html/simulate.info.html.template"));
> simulateTemplate = IOUtils.toString(
> cl.getResourceAsStream("html/simulate.html.template"));
> trackTemplate = IOUtils.toString(
> cl.getResourceAsStream("html/track.html.template"));
> {code}
> {{IOUtils.toString(InputStream, Charset)}} should be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10514) Introduce a dominant resource based schedule policy to increase the resource utilization, avoid heavy cluster resource fragments.
[ https://issues.apache.org/jira/browse/YARN-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243071#comment-17243071 ] zhuqi commented on YARN-10514: -- [~leftnoteasy] [~tangzhankun] [~prabhujoseph] [~sunil.gov...@gmail.com] [~jiwq] If you any advice about this proposal. I submit a draft patch to support async multi node scheduling mode, but i think in heartbeat schedule mode we also should to handle resource resource fragments, to help get better resource utilization. > Introduce a dominant resource based schedule policy to increase the resource > utilization, avoid heavy cluster resource fragments. > - > > Key: YARN-10514 > URL: https://issues.apache.org/jira/browse/YARN-10514 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0, 3.4.0 >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10514.001.patch > > > When we schedule in multi node lookup policy for async scheduling, or just > use heartbeat update based scheduling, we both meet scheduling fragments. > When cpu-intensive jobs or gpu-intensive or memory-intensive etc, the cluster > will meet heavy waste of resources, so this issue will help to move scheduler > support dominant resource based schedule, to help our cluster get better > resource utilization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10514) Introduce a dominant resource based schedule policy to increase the resource utilization, avoid heavy cluster resource fragments.
zhuqi created YARN-10514: Summary: Introduce a dominant resource based schedule policy to increase the resource utilization, avoid heavy cluster resource fragments. Key: YARN-10514 URL: https://issues.apache.org/jira/browse/YARN-10514 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.3.0, 3.4.0 Reporter: zhuqi Assignee: zhuqi When we schedule in multi node lookup policy for async scheduling, or just use heartbeat update based scheduling, we both meet scheduling fragments. When cpu-intensive jobs or gpu-intensive or memory-intensive etc, the cluster will meet heavy waste of resources, so this issue will help to move scheduler support dominant resource based schedule, to help our cluster get better resource utilization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Agshin Kazimli updated YARN-7200:
-
Attachment: YARN-7200.003.patch
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
>
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Grant Sohn
>Assignee: Agshin Kazimli
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch,
> YARN-7200.003.patch, snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
>
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Auto Queue Creation Check RM rest API impact
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10512: - Summary: CS Auto Queue Creation Check RM rest API impact (was: Check RM rest API impact) > CS Auto Queue Creation Check RM rest API impact > --- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10513) CS Auto Queue Creation RM UIv2 modifications
Benjamin Teke created YARN-10513: Summary: CS Auto Queue Creation RM UIv2 modifications Key: YARN-10513 URL: https://issues.apache.org/jira/browse/YARN-10513 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10511) Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
[
https://issues.apache.org/jira/browse/YARN-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243049#comment-17243049
]
Hadoop QA commented on YARN-10511:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
1s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
52m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 43s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 46s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2512/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2512 |
| JIRA Issue | YARN-10511 |
| Optional Tests | dupname asflicense mvnsite markdownlint |
| uname | Linux e121daf65ec6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 42a29199c00 |
| Max. process+thread count | 665 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2512/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update yarn.nodemanager.env-whitelist value in Hadoop Cluster Setup
> ---
>
> Key: YARN-10511
> URL: https://issues.apache.org/jira/browse/YARN-10511
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Andrea Scarpino
>Priority: Trivial
> Labels: documentation, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The [Hadoop Cluster
> Setup|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html]
> documentation page references an old value for the
> {{yarn.nodemanager.env-whitelist}} variable because the value referenced in
> that page has been changed since YARN-6694.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (YARN-10512) Check RM rest API impact
Benjamin Teke created YARN-10512: Summary: Check RM rest API impact Key: YARN-10512 URL: https://issues.apache.org/jira/browse/YARN-10512 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10031) Create a general purpose log request with additional query parameters
[
https://issues.apache.org/jira/browse/YARN-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243045#comment-17243045
]
Adam Antal commented on YARN-10031:
---
Reuploaded patch v5
> Create a general purpose log request with additional query parameters
> -
>
> Key: YARN-10031
> URL: https://issues.apache.org/jira/browse/YARN-10031
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10031-WIP.001.patch, YARN-10031.001.patch,
> YARN-10031.002.patch, YARN-10031.003.patch, YARN-10031.004.patch,
> YARN-10031.005.patch, YARN-10031.005.patch
>
>
> The current endpoints are robust but not very flexible with regards to
> filtering options. I suggest to add an endpoint which provides filtering
> options.
> E.g.:
> In ATS we have multiple endpoints:
> /containers/{containerid}/logs/{filename}
> /containerlogs/{containerid}/{filename}
> We could add @QueryParams parameters to the REST endpoints like this:
> /containers/{containerid}/logs?fileName=stderr=FAILED=nm45
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10031) Create a general purpose log request with additional query parameters
[
https://issues.apache.org/jira/browse/YARN-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10031:
--
Attachment: YARN-10031.005.patch
> Create a general purpose log request with additional query parameters
> -
>
> Key: YARN-10031
> URL: https://issues.apache.org/jira/browse/YARN-10031
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10031-WIP.001.patch, YARN-10031.001.patch,
> YARN-10031.002.patch, YARN-10031.003.patch, YARN-10031.004.patch,
> YARN-10031.005.patch, YARN-10031.005.patch
>
>
> The current endpoints are robust but not very flexible with regards to
> filtering options. I suggest to add an endpoint which provides filtering
> options.
> E.g.:
> In ATS we have multiple endpoints:
> /containers/{containerid}/logs/{filename}
> /containerlogs/{containerid}/{filename}
> We could add @QueryParams parameters to the REST endpoints like this:
> /containers/{containerid}/logs?fileName=stderr=FAILED=nm45
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9883) Reshape SchedulerHealth class
[
https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243030#comment-17243030
]
Adam Antal commented on YARN-9883:
--
Committed to trunk, thanks for the contribution [~dmmkr]
> Reshape SchedulerHealth class
> -
>
> Key: YARN-9883
> URL: https://issues.apache.org/jira/browse/YARN-9883
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: D M Murali Krishna Reddy
>Priority: Minor
> Attachments: YARN-9883.001.patch, YARN-9883.002.patch
>
>
> The {{SchedulerHealth}} class has some flaws, for example:
> - It has no javadoc at all
> - All its objects are package-private: they should be private
> - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
> are more efficient in storing Enums
> - schedulerHealthDetails only stores the last operation, its name should
> reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
