[jira] [Updated] (MAPREDUCE-7183) Make app master recover history from latest history file that exists
[ https://issues.apache.org/jira/browse/MAPREDUCE-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7183: - Attachment: MAPREDUCE-7183.patch Status: Patch Available (was: Open) > Make app master recover history from latest history file that exists > > > Key: MAPREDUCE-7183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Reporter: Mikayla Konst >Priority: Major > Attachments: MAPREDUCE-7183.patch > > > When running a mapreduce job, when the original app master is killed, the new > app master normally attempts to recover by reading the jhist file that was > written by the app master from the previous app attempt (e.g. current app > attempt - 1). > This is usually fine, but is a problem in the following situation: > # App master 1 writes history to jobid_1.jhist, then is killed > # App master 2 starts up but is killed before it has the chance to write any > history to jobid_2.jhist > # App master 3 attempts to recover, but it can't find jobid_2.jhist, so all > job progress is lost. > This problem manifests as "Unable to parse prior job history, aborting > recovery" and "Could not parse the old history file. Will not have old > AMinfos" errors, all job progress being lost, and previous app attempts not > showing up in the job history UI. > To fix this problem, if jobid_2.jhist is missing, app master 3 should just > recover using the history in jobid_1.jhist. > Related JIRAs that mention this same problem: > https://issues.apache.org/jira/browse/MAPREDUCE-4729 > https://issues.apache.org/jira/browse/MAPREDUCE-4767 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7183) Make app master recover history from latest history file that exists
Mikayla Konst created MAPREDUCE-7183: Summary: Make app master recover history from latest history file that exists Key: MAPREDUCE-7183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7183 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Mikayla Konst When running a mapreduce job, when the original app master is killed, the new app master normally attempts to recover by reading the jhist file that was written by the app master from the previous app attempt (e.g. current app attempt - 1). This is usually fine, but is a problem in the following situation: # App master 1 writes history to jobid_1.jhist, then is killed # App master 2 starts up but is killed before it has the chance to write any history to jobid_2.jhist # App master 3 attempts to recover, but it can't find jobid_2.jhist, so all job progress is lost. This problem manifests as "Unable to parse prior job history, aborting recovery" and "Could not parse the old history file. Will not have old AMinfos" errors, all job progress being lost, and previous app attempts not showing up in the job history UI. To fix this problem, if jobid_2.jhist is missing, app master 3 should just recover using the history in jobid_1.jhist. Related JIRAs that mention this same problem: https://issues.apache.org/jira/browse/MAPREDUCE-4729 https://issues.apache.org/jira/browse/MAPREDUCE-4767 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7173) Add ability to shuffle intermediate map task output to a distributed filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7173: - Attachment: MAPREDUCE-7173.patch Status: Patch Available (was: Open) > Add ability to shuffle intermediate map task output to a distributed > filesystem > --- > > Key: MAPREDUCE-7173 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7173 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 2.9.2 >Reporter: Mikayla Konst >Priority: Major > Attachments: MAPREDUCE-7173.patch > > > If nodes are lost during the course of a mapreduce job, the map tasks that > ran on those nodes need to be re-run. Writing intermediate map task output to > a distributed file system eliminates this problem in environments in which > nodes are frequently lost, for example, in clusters that make heavy use of > Google's Preemptible VMs or AWS's Spot Instances. > *Example Usage:* > *Job-scoped properties:* > 1. Don't re-run an already-finished map task when we realize the node it ran > on is now unusable: > mapreduce.map.rerun-if-node-unusable=false (see MAPREDUCE-7168) > 2. On the map side, use a new implementation of MapOutputFile that provides > paths relative to the staging dir for the job (which is cleaned up when the > job is done): > mapreduce.task.general.output.class=org.apache.hadoop.mapred.HCFSOutputFiles > 3. On the reduce side, use a new implementation of ShuffleConsumerPlugin that > fetches map task output directly from a distributed filesystem: > mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.HCFSShuffle > 4. (Optional) Edit the buffer size for the output stream used when writing > map task output > mapreduce.map.shuffle.output.buffer.size=8192 > *Cluster-scoped properties* (see YARN-9106): > 1. When gracefully decommissioning a node, only wait for the containers on > that node to finish, not the applications associated with those containers > (we don't need to wait on the applications to finish since this node is not > serving shuffle data) > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-applications=false > 2. When gracefully decommissioning a node, do not wait for app masters > running on the node to finish so that this node can be decommissioned as soon > as possible (failover to an app master on another node that isn't being > decommissioned is pretty quick) > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-app-masters=false -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7173) Add ability to shuffle intermediate map task output to a distributed filesystem
Mikayla Konst created MAPREDUCE-7173: Summary: Add ability to shuffle intermediate map task output to a distributed filesystem Key: MAPREDUCE-7173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7173 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 2.9.2 Reporter: Mikayla Konst If nodes are lost during the course of a mapreduce job, the map tasks that ran on those nodes need to be re-run. Writing intermediate map task output to a distributed file system eliminates this problem in environments in which nodes are frequently lost, for example, in clusters that make heavy use of Google's Preemptible VMs or AWS's Spot Instances. *Example Usage:* *Job-scoped properties:* 1. Don't re-run an already-finished map task when we realize the node it ran on is now unusable: mapreduce.map.rerun-if-node-unusable=false (see MAPREDUCE-7168) 2. On the map side, use a new implementation of MapOutputFile that provides paths relative to the staging dir for the job (which is cleaned up when the job is done): mapreduce.task.general.output.class=org.apache.hadoop.mapred.HCFSOutputFiles 3. On the reduce side, use a new implementation of ShuffleConsumerPlugin that fetches map task output directly from a distributed filesystem: mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.HCFSShuffle 4. (Optional) Edit the buffer size for the output stream used when writing map task output mapreduce.map.shuffle.output.buffer.size=8192 *Cluster-scoped properties* (see YARN-9106): 1. When gracefully decommissioning a node, only wait for the containers on that node to finish, not the applications associated with those containers (we don't need to wait on the applications to finish since this node is not serving shuffle data) yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-applications=false 2. When gracefully decommissioning a node, do not wait for app masters running on the node to finish so that this node can be decommissioned as soon as possible (failover to an app master on another node that isn't being decommissioned is pretty quick) yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-app-masters=false -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7171) Add additional cleanup hook to the app master
Mikayla Konst created MAPREDUCE-7171: Summary: Add additional cleanup hook to the app master Key: MAPREDUCE-7171 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7171 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Reporter: Mikayla Konst Attachments: MAPREDUCE-7171.patch Add additional cleanup hook to app master. Added property mapreduce.job.cleanup.plugins. This property is a list of cleanup plugins that should be run at the end of a job. Each plugin must implement the ApplicationCleanup interface. Plugin classes are specified using mapreduce.job.cleanup.plugins.%s.class properties. For example: mapreduce.job.cleanup.plugins=plugin1,plugin2,plugin2 mapreduce.job.cleanup.plugins.plugin1.class=CleanupClass -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7171) Add additional cleanup hook to the app master
[ https://issues.apache.org/jira/browse/MAPREDUCE-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7171: - Attachment: MAPREDUCE-7171.patch Status: Patch Available (was: Open) > Add additional cleanup hook to the app master > - > > Key: MAPREDUCE-7171 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7171 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Mikayla Konst >Priority: Minor > Attachments: MAPREDUCE-7171.patch > > > Add additional cleanup hook to app master. > Added property mapreduce.job.cleanup.plugins. This property is a list of > cleanup plugins that should be run at the end of a job. Each plugin must > implement the ApplicationCleanup interface. Plugin classes are specified > using mapreduce.job.cleanup.plugins.%s.class properties. > For example: > mapreduce.job.cleanup.plugins=plugin1,plugin2,plugin2 > mapreduce.job.cleanup.plugins.plugin1.class=CleanupClass -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7168) Add option to not kill already-done map tasks when node becomes unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7168: - Attachment: MAPREDUCE-7168.patch > Add option to not kill already-done map tasks when node becomes unusable > > > Key: MAPREDUCE-7168 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7168 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 2.9.2 > Environment: Google Compute Engine (Dataproc), Java 8 >Reporter: Mikayla Konst >Priority: Minor > Attachments: MAPREDUCE-7168.patch > > > When a node becomes unusable, if there are still reduce tasks running, all > completed map tasks that were run on that node are killed so that they can be > re-run on a different node. This is because the node can no longer serve > shuffle data, so the map task output cannot be fetched by the reducers. > If map tasks do not write their shuffle data locally, killing already-done > map tasks will make the job lose map progress unnecessarily. This change > prevents map progress from being lost when shuffle data is not written > locally by providing a property mapreduce.map.rerun-if-node-unusable that can > be set to false to prevent killing already-done map tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7168) Add option to not kill already-done map tasks when node becomes unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7168: - Attachment: (was: MAPREDUCE-7168.patch) > Add option to not kill already-done map tasks when node becomes unusable > > > Key: MAPREDUCE-7168 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7168 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 2.9.2 > Environment: Google Compute Engine (Dataproc), Java 8 >Reporter: Mikayla Konst >Priority: Minor > Attachments: MAPREDUCE-7168.patch > > > When a node becomes unusable, if there are still reduce tasks running, all > completed map tasks that were run on that node are killed so that they can be > re-run on a different node. This is because the node can no longer serve > shuffle data, so the map task output cannot be fetched by the reducers. > If map tasks do not write their shuffle data locally, killing already-done > map tasks will make the job lose map progress unnecessarily. This change > prevents map progress from being lost when shuffle data is not written > locally by providing a property mapreduce.map.rerun-if-node-unusable that can > be set to false to prevent killing already-done map tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7168) Add option to not kill already-done map tasks when node becomes unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikayla Konst updated MAPREDUCE-7168: - Attachment: MAPREDUCE-7168.patch Status: Patch Available (was: Open) > Add option to not kill already-done map tasks when node becomes unusable > > > Key: MAPREDUCE-7168 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7168 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 2.9.2 > Environment: Google Compute Engine (Dataproc), Java 8 >Reporter: Mikayla Konst >Priority: Minor > Attachments: MAPREDUCE-7168.patch > > > When a node becomes unusable, if there are still reduce tasks running, all > completed map tasks that were run on that node are killed so that they can be > re-run on a different node. This is because the node can no longer serve > shuffle data, so the map task output cannot be fetched by the reducers. > If map tasks do not write their shuffle data locally, killing already-done > map tasks will make the job lose map progress unnecessarily. This change > prevents map progress from being lost when shuffle data is not written > locally by providing a property mapreduce.map.rerun-if-node-unusable that can > be set to false to prevent killing already-done map tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7168) Add option to not kill already-done map tasks when node becomes unusable
Mikayla Konst created MAPREDUCE-7168: Summary: Add option to not kill already-done map tasks when node becomes unusable Key: MAPREDUCE-7168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7168 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 2.9.2 Environment: Google Compute Engine (Dataproc), Java 8 Reporter: Mikayla Konst When a node becomes unusable, if there are still reduce tasks running, all completed map tasks that were run on that node are killed so that they can be re-run on a different node. This is because the node can no longer serve shuffle data, so the map task output cannot be fetched by the reducers. If map tasks do not write their shuffle data locally, killing already-done map tasks will make the job lose map progress unnecessarily. This change prevents map progress from being lost when shuffle data is not written locally by providing a property mapreduce.map.rerun-if-node-unusable that can be set to false to prevent killing already-done map tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
