[jira] [Commented] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640556#comment-16640556 ] Wang Yan commented on MAPREDUCE-7148: - [~ste...@apache.org] Thanks for your comment! I like the idea of creating a general exception in hadoop-common. This is more straightforward to solve our problem. I will create the patch. > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639647#comment-16639647 ] Steve Loughran commented on MAPREDUCE-7148: --- Looks like DFS QuotaExceededException is a direct subclass of IOE. You can create a new exception in hadoop-common, like "StorageCapacityExceededException", have the DFS one subclass that. And then look for the generic one. By having it more generic than just quota, it could be used in other cases where the FS has run out of space. This would be a lot eaiser to work with than new config options, reflection games, testing thereof, etc > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639390#comment-16639390 ] Wang Yan commented on MAPREDUCE-7148: - Thanks! I will come back shortly. > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639328#comment-16639328 ] Tsuyoshi Ozawa edited comment on MAPREDUCE-7148 at 10/5/18 6:25 AM: [~tiana528] It sounds good to me specify FQCNs via configurations if it works. I made you assignee of this task. When you'are updating your patch, please add test code to verify your modification works correctly. was (Author: ozawa): [~tiana528] It sounds good to me specify FQCNs via configurations if it works. I made you assignee of this task. When you'are updating your patch, please add test code to verify your modification work correctly. > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639328#comment-16639328 ] Tsuyoshi Ozawa edited comment on MAPREDUCE-7148 at 10/5/18 6:26 AM: [~tiana528] It sounds good to me specify FQCNs via configurations if it works. I made you assignee of this task. When you'are updating your patch, please add test code to verify that your modification works correctly. was (Author: ozawa): [~tiana528] It sounds good to me specify FQCNs via configurations if it works. I made you assignee of this task. When you'are updating your patch, please add test code to verify your modification works correctly. > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned MAPREDUCE-7148: - Assignee: Wang Yan > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7148) Fast fail jobs when exceeds dfs quota limitation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639328#comment-16639328 ] Tsuyoshi Ozawa commented on MAPREDUCE-7148: --- [~tiana528] It sounds good to me specify FQCNs via configurations if it works. I made you assignee of this task. When you'are updating your patch, please add test code to verify your modification work correctly. > Fast fail jobs when exceeds dfs quota limitation > > > Key: MAPREDUCE-7148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 > Environment: hadoop 2.7.3 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Major > Attachments: MAPREDUCE-7148.001.patch > > > We are running hive jobs with a DFS quota limitation per job(3TB). If a job > hits DFS quota limitation, the task that hit it will fail and there will be a > few task reties before the job actually fails. The retry is not very helpful > because the job will always fail anyway. In some worse cases, we have a job > which has a single reduce task writing more than 3TB to HDFS over 20 hours, > the reduce task exceeds the quota limitation and retries 4 times until the > job fails in the end thus consuming a lot of unnecessary resource. This > ticket aims at providing the feature to let a job fail fast when it writes > too much data to the DFS and exceeds the DFS quota limitation. The fast fail > feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
