[jira] [Comment Edited] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199191#comment-17199191 ] zhenzhao wang edited comment on MAPREDUCE-7294 at 9/21/20, 7:18 AM: [~liuml07] Thanks a lot for reviewing. Sure, created a new PR. [https://github.com/apache/hadoop/pull/2319] Modified the patch to make it work for Java 7. was (Author: wzzdreamer): [~liuml07] Thanks a lot for reviewing. Sure, created a new PR. https://issues.apache.org/jira/browse/MAPREDUCE-7294 Modified the patch to make it work for Java 7. > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199191#comment-17199191 ] zhenzhao wang edited comment on MAPREDUCE-7294 at 9/21/20, 7:02 AM: [~liuml07] Thanks a lot for reviewing. Sure, created a new PR. https://issues.apache.org/jira/browse/MAPREDUCE-7294 Modified the patch to make it work for Java 7. was (Author: wzzdreamer): [~liuml07] Thanks a lot for reviewing. Sure, I uploaded [^MAPREDUCE-72942-2.10.001.patch] . It should work for branch 2.10, 2.10.0, and 2.10.1. > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
