[jira] [Resolved] (YARN-5853) TestDelegationTokenRenewer#testRMRestartWithExpiredToken fails intermittently on Power
[ https://issues.apache.org/jira/browse/YARN-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-5853. --- Resolution: Duplicate This is fixed by YARN-10500 > TestDelegationTokenRenewer#testRMRestartWithExpiredToken fails intermittently > on Power > -- > > Key: YARN-5853 > URL: https://issues.apache.org/jira/browse/YARN-5853 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 > Environment: # uname -a > Linux pts00452-vm10 3.10.0-327.el7.ppc64le #1 SMP Thu Oct 29 17:31:13 EDT > 2015 ppc64le ppc64le ppc64le GNU/Linux > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.2 (Maipo) >Reporter: Yussuf Shaikh >Priority: Major > > The test testRMRestartWithExpiredToken fails intermittently with the > following error: > Stacktrace: > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertNotNull(Assert.java:621) > at org.junit.Assert.assertNotNull(Assert.java:631) > at > org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testRMRestartWithExpiredToken(TestDelegationTokenRenewer.java:1060) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10500) TestDelegationTokenRenewer fails intermittently
[
https://issues.apache.org/jira/browse/YARN-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283357#comment-17283357
]
Jim Brennan commented on YARN-10500:
Thanks for the update [~iwasakims]! I have committed to trunk and will
cherry-pick to other branches.
> TestDelegationTokenRenewer fails intermittently
> ---
>
> Key: YARN-10500
> URL: https://issues.apache.org/jira/browse/YARN-10500
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> TestDelegationTokenRenewer sometimes timeouts.
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/334/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
> {noformat}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
> [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 83.675 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
> [ERROR]
> testTokenThreadTimeout(org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer)
> Time elapsed: 30.065 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 3
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout(TestDelegationTokenRenewer.java:1769)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10624) Support max leaf queues limit configuration, consistent with old auto created.
Qi Zhu created YARN-10624: - Summary: Support max leaf queues limit configuration, consistent with old auto created. Key: YARN-10624 URL: https://issues.apache.org/jira/browse/YARN-10624 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu Since old created leaf queue has the max leaf queues limit, i think we also should support this in new auto created queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280899#comment-17280899 ] Qi Zhu edited comment on YARN-10532 at 2/11/21, 3:13 PM: - [~gandras] I have updated it in testAutoCreateQueueAfterRemoval in latest patch, Thanks a lot for your patient review. cc [~wangda] [~ztang] [~epayne] [~snemeth] [~bteke] [~shuzirra] [~ebadger] Could you help review latest patch? was (Author: zhuqi): [~gandras] I have updated it in testAutoCreateQueueAfterRemoval in latest patch, Thanks a lot for your patient review. cc [~wangda] [~epayne] [~snemeth] [~bteke] [~shuzirra] [~ebadger] Could you help review latest patch? > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10610) Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath
[ https://issues.apache.org/jira/browse/YARN-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283017#comment-17283017 ] Qi Zhu commented on YARN-10610: --- !image-2021-02-11-20-58-37-436.png|width=496,height=255! I have checked in my local cluster. > Add queuePath to RESTful API for CapacityScheduler consistent with > FairScheduler queuePath > -- > > Key: YARN-10610 > URL: https://issues.apache.org/jira/browse/YARN-10610 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10610.001.patch, YARN-10610.002.patch, > YARN-10610.003.patch, image-2021-02-03-13-47-13-516.png, > image-2021-02-11-20-58-37-436.png > > > The cs only have queueName, but not full queuePath. > !image-2021-02-03-13-47-13-516.png|width=631,height=356! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10610) Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath
[ https://issues.apache.org/jira/browse/YARN-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10610: -- Attachment: image-2021-02-11-20-58-37-436.png > Add queuePath to RESTful API for CapacityScheduler consistent with > FairScheduler queuePath > -- > > Key: YARN-10610 > URL: https://issues.apache.org/jira/browse/YARN-10610 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10610.001.patch, YARN-10610.002.patch, > YARN-10610.003.patch, image-2021-02-03-13-47-13-516.png, > image-2021-02-11-20-58-37-436.png > > > The cs only have queueName, but not full queuePath. > !image-2021-02-03-13-47-13-516.png|width=631,height=356! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10588) Percentage of queue and cluster is zero in WebUI
[
https://issues.apache.org/jira/browse/YARN-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282988#comment-17282988
]
Bilwa S T commented on YARN-10588:
--
[~epayne]
Modifying *DominantResourceCalculator#isInvalidDivisor* to match logic of
*DominantResourceCalculator#divide* is nothing but returning true only if all
resource value is *0*. We already have a method called
*DominantResourceCalculator#isAllInvalidDivisor* which will return true only if
all resources are *zero*. I think we can just change isInvalidDivisor to
isAllInvalidDivisor.
> Percentage of queue and cluster is zero in WebUI
> -
>
> Key: YARN-10588
> URL: https://issues.apache.org/jira/browse/YARN-10588
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10588.001.patch, YARN-10588.002.patch,
> YARN-10588.003.patch
>
>
> Steps to reproduce:
> Configure below property in resource-types.xml
> {code:java}
>
> yarn.resource-types
> yarn.io/gpu
> {code}
> Submit a job
> In UI you can see % Of Queue and % Of Cluster is zero for the submitted
> application
>
> This is because in SchedulerApplicationAttempt has below check for
> calculating queueUsagePerc and clusterUsagePerc
> {code:java}
> if (!calc.isInvalidDivisor(cluster)) {
> float queueCapacityPerc = queue.getQueueInfo(false, false)
> .getCapacity();
> queueUsagePerc = calc.divide(cluster, usedResourceClone,
> Resources.multiply(cluster, queueCapacityPerc)) * 100;
> if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
> queueUsagePerc = 0.0f;
> }
> clusterUsagePerc =
> calc.divide(cluster, usedResourceClone, cluster) * 100;
> }
> {code}
> calc.isInvalidDivisor(cluster) always returns true as gpu resource is 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones
[ https://issues.apache.org/jira/browse/YARN-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10618: -- Fix Version/s: 3.4.0 > RM UI2 Application page shows the AM preempted containers instead of the > nonAM ones > --- > > Key: YARN-10618 > URL: https://issues.apache.org/jira/browse/YARN-10618 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10618.001.patch > > > YARN RM UIv2 application page shows the AM preempted containers under both > the _Num Non-AM container preempted_ and _Num AM container preempted_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones
[ https://issues.apache.org/jira/browse/YARN-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282919#comment-17282919 ] Szilard Nemeth commented on YARN-10618: --- Hi [~bteke], Thanks for working on this. Patch LGTM, committed to trunk. > RM UI2 Application page shows the AM preempted containers instead of the > nonAM ones > --- > > Key: YARN-10618 > URL: https://issues.apache.org/jira/browse/YARN-10618 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Attachments: YARN-10618.001.patch > > > YARN RM UIv2 application page shows the AM preempted containers under both > the _Num Non-AM container preempted_ and _Num AM container preempted_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
