[jira] [Updated] (YARN-7812) Improvements to Rich Placement Constraints in YARN
[ https://issues.apache.org/jira/browse/YARN-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-7812: -- Issue Type: Improvement (was: Bug) > Improvements to Rich Placement Constraints in YARN > -- > > Key: YARN-7812 > URL: https://issues.apache.org/jira/browse/YARN-7812 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8555) Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options
[
https://issues.apache.org/jira/browse/YARN-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593149#comment-16593149
]
Weiwei Yang edited comment on YARN-8555 at 8/27/18 3:15 AM:
Looks like processor handler still has some limitation to support some of PC
forms, such as multi allocation tags, see more in the validation code in
{{MemoryPlacementConstraintManager}}. And some of UT code in
\{{TestSchedulingRequestContainerAllocation}} triggers a CS event to update,
that doesn't work for processor path. This is causing some of UT could not pass
under the {{processor}} path. Will need to take some more time getting them
consistent.
was (Author: cheersyang):
Looks like processor handler still has some limitation to support some of PC
forms, such as multi allocation tags, see more in the validation code in
{{MemoryPlacementConstraintManager}}. This is causing some of UT could not pass
under the \{{processor}} path.
So \{{scheduler}} path gives a more complete support for various of PCs,
\{{processor}} path is more limited right now. Will need to take some more time
getting them consistent.
> Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC
> handler options
> --
>
> Key: YARN-8555
> URL: https://issues.apache.org/jira/browse/YARN-8555
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Priority: Minor
> Labels: newbie
>
> Current test cases in this 2 classes are only targeting for 1 handler type,
> \{{scheduler}} or \{{processor}}. Once YARN-8015 is done, we should modify
> them to be parameterized in order to cover both cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8555) Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options
[
https://issues.apache.org/jira/browse/YARN-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593149#comment-16593149
]
Weiwei Yang edited comment on YARN-8555 at 8/27/18 3:09 AM:
Looks like processor handler still has some limitation to support some of PC
forms, such as multi allocation tags, see more in the validation code in
{{MemoryPlacementConstraintManager}}. This is causing some of UT could not pass
under the \{{processor}} path.
So \{{scheduler}} path gives a more complete support for various of PCs,
\{{processor}} path is more limited right now. Will need to take some more time
getting them consistent.
was (Author: cheersyang):
Looks like processor handler still has some limitation to support some of PC
forms, such as multi allocation tags, see more in the validation code in
\{{MemoryPlacementConstraintManager}}.
> Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC
> handler options
> --
>
> Key: YARN-8555
> URL: https://issues.apache.org/jira/browse/YARN-8555
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Priority: Minor
> Labels: newbie
>
> Current test cases in this 2 classes are only targeting for 1 handler type,
> \{{scheduler}} or \{{processor}}. Once YARN-8015 is done, we should modify
> them to be parameterized in order to cover both cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8555) Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options
[
https://issues.apache.org/jira/browse/YARN-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593149#comment-16593149
]
Weiwei Yang commented on YARN-8555:
---
Looks like processor handler still has some limitation to support some of PC
forms, such as multi allocation tags, see more in the validation code in
\{{MemoryPlacementConstraintManager}}.
> Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC
> handler options
> --
>
> Key: YARN-8555
> URL: https://issues.apache.org/jira/browse/YARN-8555
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Priority: Minor
> Labels: newbie
>
> Current test cases in this 2 classes are only targeting for 1 handler type,
> \{{scheduler}} or \{{processor}}. Once YARN-8015 is done, we should modify
> them to be parameterized in order to cover both cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8717) set memory.limit_in_bytes when NodeManager starting
[ https://issues.apache.org/jira/browse/YARN-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-8717: Environment: (was: CGroupsCpuResourceHandlerImpl sets cpu quota at hirarchy of hadoop-yarn to restrict total resource of cpu of NM when NM starting; CGroupsMemoryResourceHandlerImpl also should set memory.limit_in_bytes at hirachy of hadoop-yarn to control cpu resource of NM) > set memory.limit_in_bytes when NodeManager starting > --- > > Key: YARN-8717 > URL: https://issues.apache.org/jira/browse/YARN-8717 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8717) set memory.limit_in_bytes when NodeManager starting
Jiandan Yang created YARN-8717: --- Summary: set memory.limit_in_bytes when NodeManager starting Key: YARN-8717 URL: https://issues.apache.org/jira/browse/YARN-8717 Project: Hadoop YARN Issue Type: New Feature Environment: CGroupsCpuResourceHandlerImpl sets cpu quota at hirarchy of hadoop-yarn to restrict total resource of cpu of NM when NM starting; CGroupsMemoryResourceHandlerImpl also should set memory.limit_in_bytes at hirachy of hadoop-yarn to control cpu resource of NM Reporter: Jiandan Yang Assignee: Jiandan Yang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8717) set memory.limit_in_bytes when NodeManager starting
[ https://issues.apache.org/jira/browse/YARN-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-8717: Description: CGroupsCpuResourceHandlerImpl sets cpu quota at hirarchy of hadoop-yarn to restrict total resource of cpu of NM when NM starting; CGroupsMemoryResourceHandlerImpl also should set memory.limit_in_bytes at hirachy of hadoop-yarn to control cpu resource of NM > set memory.limit_in_bytes when NodeManager starting > --- > > Key: YARN-8717 > URL: https://issues.apache.org/jira/browse/YARN-8717 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > CGroupsCpuResourceHandlerImpl sets cpu quota at hirarchy of hadoop-yarn to > restrict total resource of cpu of NM when NM starting; > CGroupsMemoryResourceHandlerImpl also should set memory.limit_in_bytes at > hirachy of hadoop-yarn to control cpu resource of NM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized
[
https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593141#comment-16593141
]
niu commented on YARN-8513:
---
OK. I will try.
> CapacityScheduler infinite loop when queue is near fully utilized
> -
>
> Key: YARN-8513
> URL: https://issues.apache.org/jira/browse/YARN-8513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 3.1.0, 2.9.1
> Environment: Ubuntu 14.04.5 and 16.04.4
> YARN is configured with one label and 5 queues.
>Reporter: Chen Yufei
>Priority: Major
> Attachments: jstack-1.log, jstack-2.log, jstack-3.log, jstack-4.log,
> jstack-5.log, top-during-lock.log, top-when-normal.log, yarn3-jstack1.log,
> yarn3-jstack2.log, yarn3-jstack3.log, yarn3-jstack4.log, yarn3-jstack5.log,
> yarn3-resourcemanager.log, yarn3-top
>
>
> ResourceManager does not respond to any request when queue is near fully
> utilized sometimes. Sending SIGTERM won't stop RM, only SIGKILL can. After RM
> restart, it can recover running jobs and start accepting new ones.
>
> Seems like CapacityScheduler is in an infinite loop printing out the
> following log messages (more than 25,000 lines in a second):
>
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> assignedContainer queue=root usedCapacity=0.99816763
> absoluteUsedCapacity=0.99816763 used=
> cluster=}}
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Failed to accept allocation proposal}}
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
> assignedContainer application attempt=appattempt_1530619767030_1652_01
> container=null
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@14420943
> clusterResource= type=NODE_LOCAL
> requestedPartition=}}
>
> I encounter this problem several times after upgrading to YARN 2.9.1, while
> the same configuration works fine under version 2.7.3.
>
> YARN-4477 is an infinite loop bug in FairScheduler, not sure if this is a
> similar problem.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8700) Application cannot un-registered
[ https://issues.apache.org/jira/browse/YARN-8700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fox resolved YARN-8700. --- Resolution: Fixed > Application cannot un-registered > > > Key: YARN-8700 > URL: https://issues.apache.org/jira/browse/YARN-8700 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.3 >Reporter: fox >Priority: Major > > Dear all, > I found a problem with application unregistration in AWS EMR environment > (emr-5.8.0, hadoop 2.7.3, spark 2.2.0). > Application Type: Both Yarn and Spark > State: RUNNING > Inside the job logs, I got > 07:00:07.190 [main] INFO c.w.c.e.a.n.b.AbstractNormalBatchMain - [EDP2] Ready > to run Tear Down > 07:00:07.192 [main] INFO c.w.c.e.a.n.b.AbstractNormalBatchMain - [EDP2] Ready > to run Tear Down > 07:00:07.192 [main] INFO c.w.c.e.a.n.b.AbstractNormalBatchMain - [EDP2] Job > Finish > 07:00:07.195 [main] INFO o.s.c.a.AnnotationConfigApplicationContext - Closing > org.springframework.context.annotation.AnnotationConfigApplicationContext@144ab54: > startup date [Tue Aug 21 06:59:23 UTC 2018]; root of context hierarchy > 07:00:07.306 [main] INFO o.s.s.c.ThreadPoolTaskExecutor - Shutting down > ExecutorService 'redisClusterExecutor' > 07:00:07.551 [main] INFO o.a.k.clients.producer.KafkaProducer - Closing the > Kafka producer with timeoutMillis = 9223372036854775807 ms. > 07:00:07.565 [main] INFO c.w.c.f.m.MessageQueueKafkaProducerImpl - Closed all > the producer's connections for tenant: 7fd0356c-1258-11e8-abfd-0242ac110002. > 07:00:09.869 [main] INFO c.w.c.edp2.normal.batch.AppMaster - finish run main > method > 07:00:09.870 [main] INFO c.w.c.edp2.normal.batch.AppMaster - delete temp file > /tmp/aa33f388-f591-44a8-9aa3-13e2f8427c5d2802069659156113885.jar > 07:00:10.112 [main] INFO o.a.h.y.c.api.impl.AMRMClientImpl - Waiting for > application to be successfully unregistered. > 07:00:10.215 [main] INFO o.a.h.y.c.api.impl.AMRMClientImpl - Waiting for > application to be successfully unregistered. > 07:00:10.319 [main] INFO o.a.h.y.c.api.impl.AMRMClientImpl - Waiting for > application to be successfully unregistered. > 07:00:10.422 [main] INFO o.a.h.y.c.api.impl.AMRMClientImpl - Waiting for > application to be successfully unregistered. > 07:00:10.528 [main] INFO o.a.h.y.c.api.impl.AMRMClientImpl - Waiting for > application to be successfully unregistered. > > and it keeps more than one day until I stopped the whole cluster. > I also try to kill the application by yarn command, which also keeps forever > waiting for application to be killed. > hadoop@ip-10-100-2-124 ~]$ yarn application -kill > application_1534810852740_0721 > 18/08/22 12:24:28 INFO impl.TimelineClientImpl: Timeline service address: > http://ip-10-100-2-124.ap-northeast-1.compute.internal:8188/ws/v1/timeline/ > 18/08/22 12:24:29 INFO client.RMProxy: Connecting to ResourceManager at > ip-10-100-2-124.ap-northeast-1.compute.internal/10.100.2.124:8032 > Killing application application_1534810852740_0721 > 18/08/22 12:24:32 INFO impl.YarnClientImpl: Waiting for application > application_1534810852740_0721 to be killed. > 18/08/22 12:24:34 INFO impl.YarnClientImpl: Waiting for application > application_1534810852740_0721 to be killed. > 18/08/22 12:24:36 INFO impl.YarnClientImpl: Waiting for application > application_1534810852740_0721 to be killed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593113#comment-16593113 ] Weiwei Yang commented on YARN-7863: --- Sounds good to me, +1. Lets get this one in. Thanks [~sunilg], [~Naganarasimha]! > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593114#comment-16593114 ] Naganarasimha G R commented on YARN-7863: - Yes [~sunilg], as both the issues are related and we would require a test case for the weiwei's scenario too, hence i agree can be covered in other jira. Going ahead and committing the patch ! > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593112#comment-16593112 ] Sunil Govindan commented on YARN-7863: -- Thanks [~cheersyang] and [~Naganarasimha] This comment is valid and need to add more test cases. I ll take a stab in next patch with test case given the size and complexity of this patch. Thoughts? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109 ] Naganarasimha G R edited comment on YARN-7863 at 8/27/18 1:58 AM: -- Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there for scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. was (Author: naganarasimha): Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there from scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there from scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593103#comment-16593103 ] Sunil Govindan commented on YARN-7863: -- Thanks [~cheersyang] Since the constraint expression asks for java!=1.8, there could be some nodes which doesnt have java. But may nt sure abt version. Thats why i took a tighter approach for accuracy. Ur thoughts? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593099#comment-16593099 ] Weiwei Yang commented on YARN-7863: --- Hi [~sunilg] Latest patch seems good to me, except one doubt. If I specify PC as "java!=1.8", for a node that doesn't have node-attribute "java", will this node satisfy the PC or not? I am expecting a true but seems the patch will return false. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8716) [Submarine] Support passing Kerberos principle tokens when launch training jobs.
Wangda Tan created YARN-8716: Summary: [Submarine] Support passing Kerberos principle tokens when launch training jobs. Key: YARN-8716 URL: https://issues.apache.org/jira/browse/YARN-8716 Project: Hadoop YARN Issue Type: Sub-task Components: submarine Reporter: Wangda Tan -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593012#comment-16593012
]
genericqa commented on YARN-7863:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
19s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
21s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 26s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 268 unchanged - 1 fixed = 270 total (was 269) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 44s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 52s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}188m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.applications.distributedshell.TestDistributedShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-7
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592955#comment-16592955 ] Sunil Govindan commented on YARN-7863: -- Thanks [~Naganarasimha]. Updating latest patch addressing comments > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-7863: - Attachment: YARN-7863-YARN-3409.011.patch > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7086) Release all containers aynchronously
[ https://issues.apache.org/jira/browse/YARN-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7086: --- Attachment: YARN-7086.002.patch > Release all containers aynchronously > > > Key: YARN-7086 > URL: https://issues.apache.org/jira/browse/YARN-7086 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Manikandan R >Priority: Major > Attachments: YARN-7086.001.patch, YARN-7086.002.patch > > > We have noticed in production two situations that can cause deadlocks and > cause scheduling of new containers to come to a halt, especially with regard > to applications that have a lot of live containers: > # When these applicaitons release these containers in bulk. > # When these applications terminate abruptly due to some failure, the > scheduler releases all its live containers in a loop. > To handle the issues mentioned above, we have a patch in production to make > sure ALL container releases happen asynchronously - and it has served us well. > Opening this JIRA to gather feedback on if this is a good idea generally (cc > [~leftnoteasy], [~jlowe], [~curino], [~kasha], [~subru], [~roniburd]) > BTW, In YARN-6251, we already have an asyncReleaseContainer() in the > AbstractYarnScheduler and a corresponding scheduler event, which is currently > used specifically for the container-update code paths (where the scheduler > realeases temp containers which it creates for the update) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7086) Release all containers aynchronously
[ https://issues.apache.org/jira/browse/YARN-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592867#comment-16592867 ] Manikandan R commented on YARN-7086: [~jlowe] Thanks for sharing the background details. Attached .002 patch to contain the changes required to acquire LeafQueue lock only once to release set of containers. Introduced wrapper methods on top of existing methods to re use the functionality wherever possible. On the flip side, ended up in traversing same set of containers for processing few more times. Please review and share your comments. If approach is fine, can drill down more to see for any further improvements. > Release all containers aynchronously > > > Key: YARN-7086 > URL: https://issues.apache.org/jira/browse/YARN-7086 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Manikandan R >Priority: Major > Attachments: YARN-7086.001.patch > > > We have noticed in production two situations that can cause deadlocks and > cause scheduling of new containers to come to a halt, especially with regard > to applications that have a lot of live containers: > # When these applicaitons release these containers in bulk. > # When these applications terminate abruptly due to some failure, the > scheduler releases all its live containers in a loop. > To handle the issues mentioned above, we have a patch in production to make > sure ALL container releases happen asynchronously - and it has served us well. > Opening this JIRA to gather feedback on if this is a good idea generally (cc > [~leftnoteasy], [~jlowe], [~curino], [~kasha], [~subru], [~roniburd]) > BTW, In YARN-6251, we already have an asyncReleaseContainer() in the > AbstractYarnScheduler and a corresponding scheduler event, which is currently > used specifically for the container-update code paths (where the scheduler > realeases temp containers which it creates for the update) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592847#comment-16592847 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], Latest patch does not seem to consider all my [comments|https://issues.apache.org/jira/browse/YARN-7863?focusedCommentId=16589559&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16589559] and also latest patch seems to have some checkstyle issues to be addressed. Can you please take a look at them? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592827#comment-16592827
]
genericqa commented on YARN-7863:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
27s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
3s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
3s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 27s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 268 unchanged - 1 fixed = 286 total (was 269) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
22s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 4s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 2s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.applications.distributedshell.TestDistributedShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-
