[jira] [Commented] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200360#comment-17200360 ] Mingliang Liu commented on MAPREDUCE-7294: -- Thanks [~iwasakims]. I saw you concluded the vote, but did not see the notice of RC. So I asked. Glad to know. Let's target 2.10.2 on this. > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200339#comment-17200339 ] Masatake Iwasaki commented on MAPREDUCE-7294: - [~liuml07] the artifacts of 2.10.1 were already published. I'm updating site docs now. This should be targeted to 2.10.2. > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved MAPREDUCE-7294. -- Fix Version/s: 2.10.2 Resolution: Fixed Committed to all fix versions. Thanks for your contribution, [~wzzdreamer]. Thanks for your discussion and review [~jiwq] [~iwasakims] Can we have this in Hadoop 2.10.1 release? Is that frozen for commits? Thanks! > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200190#comment-17200190 ] Devaraj Kavali commented on MAPREDUCE-6749: --- Thanks [~brahmareddy] for checking on this. There are some more tasks need to be done for this feature, I am occupied with some other things, please free to take it up these if you are interested. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200059#comment-17200059 ] Brahma Reddy Battula commented on MAPREDUCE-6749: - [~devaraj] looks no progress on this, we would like know the status and work on this. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default
[ https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7282: -- Summary: MR v2 commit algorithm should be deprecated and not the default (was: MR v2 commit algorithm is dangerous, should be deprecated and not the default) > MR v2 commit algorithm should be deprecated and not the default > --- > > Key: MAPREDUCE-7282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1 >Reporter: Steve Loughran >Priority: Major > > The v2 MR commit algorithm moves files from the task attempt dir into the > dest dir on task commit -one by one > It is therefore not atomic > # if a task commit fails partway through and another task attempt commits > -unless exactly the same filenames are used, output of the first attempt may > be included in the final result > # if a worker partitions partway through task commit, and then continues > after another attempt has committed, it may partially overwrite the output > -even when the filenames are the same > Both MR and spark assume that task commits are atomic. Either they need to > consider that this is not the case, we add a way to probe for a committer > supporting atomic task commit, and the engines both add handling for task > commit failures (probably fail job) > Better: we remove this as the default, maybe also warn when it is being used -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
