[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656566#comment-16656566 ]
Steve Loughran commented on MAPREDUCE-7148: ------------------------------------------- As I've warned, I can't do the final vote as someone who knows MR needs to be the one there. Regarding the bits I do know * that new exception should be tagged @Evolving, as tagging it as @Stable boxes us out of making changes. Maybe even @LimitedPrivate(HDFS, MapReduce) for now with an explanation of what it does (e.g point at this JIRA and say "raised by HDFS to say what it does" * I like what you've done with the tests. There's now a fair bit of redundant setup for each one. If the task & umbilical were made fields, the test {{setup()}} method could do the basic init here, so cut four identical lines from each test: {code} 108 Task task = Mockito.mock(Task.class); 109 TaskUmbilicalProtocol umbilical = Mockito.mock(TaskUmbilicalProtocol.class); 110 Configuration conf = new Configuration(); 111 when(task.getConf()).thenReturn(conf); {code} > Fast fail jobs when exceeds dfs quota limitation > ------------------------------------------------ > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task > Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 > Reporter: Wang Yan > Assignee: Wang Yan > Priority: Major > Attachments: MAPREDUCE-7148.001.patch, MAPREDUCE-7148.002.patch, > MAPREDUCE-7148.003.patch, MAPREDUCE-7148.004.patch, MAPREDUCE-7148.005.patch, > MAPREDUCE-7148.006.patch, MAPREDUCE-7148.007.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org