[jira] [Issue Comment Deleted] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Comment: was deleted
(was: Hi Steve, I have used the findbugs to analyze Slider, and it seems there
are some possible bugs)
> A better way to deal with InterruptedException on waitForCompletion
> ---
>
> Key: MAPREDUCE-6296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Yang Hao
>Assignee: Yang Hao
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-6296.patch
>
>
> Some code in method waitForCompletion of Job class is
> {code:title=Job.java|borderStyle=solid}
> public boolean waitForCompletion(boolean verbose
>) throws IOException, InterruptedException,
> ClassNotFoundException {
> if (state == JobState.DEFINE) {
> submit();
> }
> if (verbose) {
> monitorAndPrintJob();
> } else {
> // get the completion poll interval from the client.
> int completionPollIntervalMillis =
> Job.getCompletionPollInterval(cluster.getConf());
> while (!isComplete()) {
> try {
> Thread.sleep(completionPollIntervalMillis);
> } catch (InterruptedException ie) {
> }
> }
> }
> return isSuccessful();
> }
> {code}
> but a better way to deal with InterruptException is
> {code:title=Job.java|borderStyle=solid}
> public boolean waitForCompletion(boolean verbose
>) throws IOException, InterruptedException,
> ClassNotFoundException {
> if (state == JobState.DEFINE) {
> submit();
> }
> if (verbose) {
> monitorAndPrintJob();
> } else {
> // get the completion poll interval from the client.
> int completionPollIntervalMillis =
> Job.getCompletionPollInterval(cluster.getConf());
> while (!isComplete()) {
> try {
> Thread.sleep(completionPollIntervalMillis);
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> }
> }
> }
> return isSuccessful();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6374) Distributed Cache File visibility should check permission of full path
[ https://issues.apache.org/jira/browse/MAPREDUCE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562049#comment-14562049 ] Yang Hao commented on MAPREDUCE-6374: - can you use submit to see whether it works? Distributed Cache File visibility should check permission of full path -- Key: MAPREDUCE-6374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6374 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6374.1.patch should do full ancestor permission check for a relative cache file input -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) Provide option to suppress stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510686#comment-14510686 ] Yang Hao commented on MAPREDUCE-6220: - Hi, thanks for your suggestion. Is there a plan to run Hadoop on Windows? Provide option to suppress stdout of MapReduce task --- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) Provide option to suppress stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492012#comment-14492012 ] Yang Hao commented on MAPREDUCE-6220: - It has been convinced that it doesn't affect the log aggregation Provide option to suppress stdout of MapReduce task --- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) Provide option to suppress stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491309#comment-14491309 ] Yang Hao commented on MAPREDUCE-6220: - I think we can change the word from forbid to suppress as you said, for that to suppress the stdout is the goal and devnull is a method to ahieve this goal. Users may be concerned on the function than the implement method Provide option to suppress stdout of MapReduce task --- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Provide option to suppress stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Summary: Provide option to suppress stdout of MapReduce task (was: Provide option to suppress stdout of MapReduce task) Provide option to suppress stdout of MapReduce task --- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Provide option to suppress stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Summary: Provide option to suppress stdout of MapReduce task (was: Provide option to forbid stdout of MapReduce task) Provide option to suppress stdout of MapReduce task Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Add an parameter for users decide whether forbid stdout of map task or reduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Description: System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it (was: System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can providing an parameter to forbid it) Add an parameter for users decide whether forbid stdout of map task or reduce task -- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486935#comment-14486935
]
Yang Hao commented on MAPREDUCE-6296:
-
Hi Steve, I have used the findbugs to analyze Slider, and it seems there are
some possible bugs
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reopened MAPREDUCE-6220: - To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Add an parameter for users decide whether forbid stdout of map task or reduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Description: System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can providing an parameter to forbid it (was: System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it) Add an parameter for users decide whether forbid stdout of map task or reduce task -- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can providing an parameter to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486999#comment-14486999 ] Yang Hao commented on MAPREDUCE-6220: - Now, I have understood that that the stderr is very helpful. How about add an parameter if the task's standout can be forbid To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Attachment: MAPREDUCE-6220.v2.patch To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Add an parameter for users decide whether forbid stdout of map task or reduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Summary: Add an parameter for users decide whether forbid stdout of map task or reduce task (was: To forbid stderr and stdout for MapReduce job) Add an parameter for users decide whether forbid stdout of map task or reduce task -- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao resolved MAPREDUCE-6176. - Resolution: Duplicate Target Version/s: (was: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Provide option to forbid stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Summary: Provide option to forbid stdout of MapReduce task (was: Add an parameter for users decide whether forbid stdout of map task or reduce task) Provide option to forbid stdout of MapReduce task -- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) Provide option to forbid stdout of MapReduce task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Affects Version/s: (was: 2.6.0) Status: Patch Available (was: Reopened) Provide option to forbid stdout of MapReduce task -- Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch, MAPREDUCE-6220.v2.patch System.out is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can provide an option to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6178) MRAppBenchmark.benchmark1() error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao resolved MAPREDUCE-6178. - Resolution: Cannot Reproduce MRAppBenchmark.benchmark1() error - Key: MAPREDUCE-6178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6178 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.1 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6178-branch2.4.1.patch when running the test, it ouputs an exception:java.lang.NullPointerException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Attachment: MAPREDUCE-6296.patch
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Status: Patch Available (was: Open)
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Description:
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
was:
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
return isSuccessful();
}
{code}
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Description:
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
return isSuccessful();
}
{code}
was:
Some code in method waitForCompletion of Job class is
{quote}
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
return isSuccessful();
}
{code}
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Description:
Some code in method waitForCompletion of Job class is
{quote}
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
return isSuccessful();
}
{code}
was:
Some code in method waitForCompletion of Job class is
{quote}
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
{quote}
but a better way to deal with InterruptException is
{quote}
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
{quote}
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Some code in method waitForCompletion of Job class is
{quote}
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Description:
Some code in method waitForCompletion of Job class is
{quote}
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
{quote}
but a better way to deal with InterruptException is
{quote}
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
{quote}
was:
Some code in method waitForCompletion of Job class is
{quote}
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
{quote}
but a better way to deal with InterruptException is
{quote}
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
{auote}
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Some code in method waitForCompletion of Job class is
{quote}
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
{quote}
but a better way to deal with InterruptException is
{quote}
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
Yang Hao created MAPREDUCE-6296:
---
Summary: A better way to deal with InterruptedException on
waitForCompletion
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Some code in method waitForCompletion of Job class is
{quote}
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
{quote}
but a better way to deal with InterruptException is
{quote}
try{
while (!isComplete()) {
Thread.sleep(completionPollIntervalMillis);
}
}catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
{auote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Attachment: MAPREDUCE-5583v3.patch Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.1.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch, MAPREDUCE-5583v3.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Attachment: (was: MAPREDUCE-5583v3.patch) Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.1.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Affects Version/s: (was: 2.3.0) Status: Open (was: Patch Available) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.6.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305036#comment-14305036 ] Yang Hao commented on MAPREDUCE-5718: - Hi, I have read the comment twice. It may not be safe to recover under this situation. I'm wondering whether you are worring about missing data. But during the recovery, task output data will be moved to the new output dir, just as the normal AM failover does, then it's safe to recover Our cluster will add this feature. Can you give more specific information on the unsafe situation? Thanks a lot MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0, 2.6.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305035#comment-14305035 ] Yang Hao commented on MAPREDUCE-5718: - Hi, I have read the comment twice. It may not be safe to recover under this situation. I'm wondering whether you are worring about missing data. But during the recovery, task output data will be moved to the new output dir, just as the normal AM failover does, then it's safe to recover Our cluster will add this feature. Can you give more specific information on the unsafe situation? Thanks a lot MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0, 2.6.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: MAPREDUCE-6232.v2.patch Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: (was: MAPREDUCE-6232.v2.patch) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: MAPREDUCE-6232.v2.patch Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Fix Version/s: 2.6.0 Target Version/s: 2.6.0 Status: Patch Available (was: Open) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Fix Version/s: 2.6.0 Labels: (was: ha) Affects Version/s: 2.6.0 Status: Patch Available (was: Reopened) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.6.0, 2.3.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302863#comment-14302863 ] Yang Hao commented on MAPREDUCE-6220: - I'm wondering in which case the oozie may use the stderr or stdout, can you give an example? To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: MAPREDUCE-6232.patch Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, result.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: (was: NewStateTransitionDiagramOfTaskImpl.gv) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, result.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6232) Task state is running when all task attempts fail
Yang Hao created MAPREDUCE-6232: --- Summary: Task state is running when all task attempts fail Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: NewStateTransitionDiagramOfTaskImpl.gv result.png Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: NewStateTransitionDiagramOfTaskImpl.gv, result.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reassigned MAPREDUCE-6232: --- Assignee: Yang Hao Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: NewStateTransitionDiagramOfTaskImpl.gv, result.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: TaskImpl.normal.png TaskImpl.new.png Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, TaskImpl.new.png, TaskImpl.normal.png, result.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: (was: result.png) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, TaskImpl.new.png, TaskImpl.normal.png When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: result.pdf Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298000#comment-14298000 ] Yang Hao commented on MAPREDUCE-6232: - I have done an experiment in pseudo-distributed env, it will work. I think you can have look at the patch to check whether it's Okay Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6229) finished MapReduce tasks will be re executed when AM fails on recovery process
[ https://issues.apache.org/jira/browse/MAPREDUCE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6229: Summary: finished MapReduce tasks will be re executed when AM fails on recovery process (was: finished MapReduce tasks may be re executed when AM fails on recovery process) finished MapReduce tasks will be re executed when AM fails on recovery process -- Key: MAPREDUCE-6229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6229 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Yang Hao MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, many tasks will be reexecuted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6229) finished MapReduce tasks may be re executed when AM fails on recovery process
Yang Hao created MAPREDUCE-6229: --- Summary: finished MapReduce tasks may be re executed when AM fails on recovery process Key: MAPREDUCE-6229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6229 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Yang Hao MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, many tasks will be reexecuted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6229) finished MapReduce tasks will be re executed when AM fails on recovery process
[ https://issues.apache.org/jira/browse/MAPREDUCE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6229: Description: MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, * many tasks will be reexecuted. * users will not see the history, such as the tasks killed or finished at the first attempt. was:MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, many tasks will be reexecuted. finished MapReduce tasks will be re executed when AM fails on recovery process -- Key: MAPREDUCE-6229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6229 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Yang Hao MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, * many tasks will be reexecuted. * users will not see the history, such as the tasks killed or finished at the first attempt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6229) finished MapReduce tasks will be re executed when AM fails on recovery process
[ https://issues.apache.org/jira/browse/MAPREDUCE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294515#comment-14294515 ] Yang Hao commented on MAPREDUCE-6229: - The history file is there in hdfs, and we can recover these information. Reduce output will always be right if done, and if some reduce tasks have not finished and some map tasks failed for nodemanager crashed, the reduce will tell AM that it cannot get data from map tasks, so the map will be reexecuted. finished MapReduce tasks will be re executed when AM fails on recovery process -- Key: MAPREDUCE-6229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6229 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Yang Hao MapReduce AM failovers, and suddently it is killed during recovery. Next time when the AM failovers again, the progress will lose for that it only parses previous history file. As a result, * many tasks will be reexecuted. * users will not see the history, such as the tasks killed or finished at the first attempt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reassigned MAPREDUCE-5718: --- Assignee: Yang Hao (was: Karthik Kambatla) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Yang Hao Labels: ha Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Attachment: MAPREDUCE-5718.v2.patch I think we can delete the commit tmp file at init stage MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Summary: MR job will fail after commit fail (was: MR job fails during commit fail) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job fails during commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Description: when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. was:While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. MR job fails during commit fail --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291561#comment-14291561 ] Yang Hao commented on MAPREDUCE-5718: - if crashed during a commit and if an api comes out to check if the job can fail over, then the problem will be fixed MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reopened MAPREDUCE-5718: - MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR AM should tolerate RM restart/failover during commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291542#comment-14291542 ] Yang Hao commented on MAPREDUCE-5718: - This will also happen when the AM is preempted MR AM should tolerate RM restart/failover during commit --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR AM should tolerate RM restart/failover during commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291550#comment-14291550 ] Yang Hao commented on MAPREDUCE-5718: - Yes, it appears when we test preempting, and the job fails for We crashed durring a commit MR AM should tolerate RM restart/failover during commit --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job fails during commit fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Summary: MR job fails during commit fails (was: MR AM should tolerate RM restart/failover during commit) MR job fails during commit fails Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job fails during commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Summary: MR job fails during commit fail (was: MR job fails during commit fails) MR job fails during commit fail --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR AM should tolerate RM restart/failover during commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291543#comment-14291543 ] Yang Hao commented on MAPREDUCE-5718: - This will also happen when the AM is preempted MR AM should tolerate RM restart/failover during commit --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
Yang Hao created MAPREDUCE-6220: --- Summary: To forbid stderr and stdout for MapReduce job Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So an alternative is to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Attachment: MAPREDUCE-6220.patch To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So an alternative is to forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6220: Description: System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it (was: System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So an alternative is to forbid it) To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reassigned MAPREDUCE-6220: --- Assignee: Yang Hao To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237385#comment-14237385 ] Yang Hao commented on MAPREDUCE-5583: - The configure mapreduce.job.running.reduces.limit in mapred-default.xml should be mapreduce.job.running.reduce.limit Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.1.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5684) TestMRJobs.testFailingMapper occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-5684:
Affects Version/s: (was: 2.2.0)
2.7.0
TestMRJobs.testFailingMapper occasionally fails
---
Key: MAPREDUCE-5684
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5684
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Liyin Liang
Attachments: MAPREDUCE-5684-1.diff
TestMRJobs is occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.mapreduce.v2.TestMRJobs
---
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 323.503 sec
FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
testFailingMapper(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time elapsed:
15.657 sec FAILURE!
java.lang.AssertionError: expected:TIPFAILED but was:FAILED
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.testFailingMapper(TestMRJobs.java:313)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6178) MRAppBenchmark.benchmark1() error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6178: Attachment: MAPREDUCE-6178-branch2.4.1.patch MRAppBenchmark.benchmark1() error - Key: MAPREDUCE-6178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6178 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.1 Reporter: Yang Hao Attachments: MAPREDUCE-6178-branch2.4.1.patch when running the test, it ouputs an exception:java.lang.NullPointerException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6178) MRAppBenchmark.benchmark1() error
Yang Hao created MAPREDUCE-6178: --- Summary: MRAppBenchmark.benchmark1() error Key: MAPREDUCE-6178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6178 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.1 Reporter: Yang Hao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Attachment: MAPREDUCE-5583-branch2.4.1.patch the patch for branch2.4.1 Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.1.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6178) MRAppBenchmark.benchmark1() error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reassigned MAPREDUCE-6178: --- Assignee: Yang Hao MRAppBenchmark.benchmark1() error - Key: MAPREDUCE-6178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6178 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.1 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6178-branch2.4.1.patch when running the test, it ouputs an exception:java.lang.NullPointerException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6178) MRAppBenchmark.benchmark1() error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6178: Description: when running the test, it ouputs an exception:java.lang.NullPointerException MRAppBenchmark.benchmark1() error - Key: MAPREDUCE-6178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6178 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.1 Reporter: Yang Hao when running the test, it ouputs an exception:java.lang.NullPointerException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Attachment: (was: MAPREDUCE-5583-branch2.4.0.patch) Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.1.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: MAPREDUCE-6176-branch2.4.0.patch To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch, MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: (was: MAPREDUCE-6176.patch) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Affects Version/s: 2.4.0 To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Target Version/s: 2.5.2, 2.5.1, 2.4.1, 2.5.0, 2.4.0 (was: 2.5.0, 2.4.1, 2.5.1, 2.5.2) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6176 started by Yang Hao. --- To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6176 stopped by Yang Hao. --- To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: (was: MAPREDUCE-6176-branch2.4.0.patch) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176-branch2.4.0.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Attachment: MAPREDUCE-5583-branch2.4.0.patch Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.0.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5583: Target Version/s: 2.4.0, 2.7.0 (was: 2.7.0) Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5583-branch2.4.0.patch, MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: (was: MAPREDUCE-6176.patch) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: MAPREDUCE-6176.patch add test To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Fix Version/s: (was: 2.4.1) 2.5.0 To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.5.0 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Fix Version/s: (was: 2.5.0) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6176) Users should limit the number of an application
Yang Hao created MAPREDUCE-6176: --- Summary: Users should limit the number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.2, 2.5.1, 2.4.1, 2.5.0 Reporter: Yang Hao Assignee: Yang Hao As MapReduce is batch framework of calculation, so people may want to run application A as well as application B . A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6176:
Release Note: (was: diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
index 7c18f06..6f69168 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
@@ -111,16 +111,18 @@
completed - request corresponding to which container has completed
Lifecycle of map
- scheduled-assigned-completed
-
+ pending-scheduled-assigned-completed
+
Lifecycle of reduce
pending-scheduled-assigned-completed
-
- Maps are scheduled as soon as their requests are received. Reduces are
- added to the pending and are ramped up (added to scheduled) based
- on completed maps and current availability in the cluster.
+
+ Maps are added to the pending are scheduled when the assigned maps plus
scheduled map less than max number of map. Reduces are
+ added to the pending and are ramped up (added to scheduled) based
+ on completed maps and current availability in the cluster as well as the max
number of reduce.
*/
-
+ //maps which are not yet scheduled
+ private final LinkedListContainerRequestEvent pendingMaps =
+new LinkedListContainerRequestEvent();
//reduces which are not yet scheduled
private final LinkedListContainerRequest pendingReduces =
new LinkedListContainerRequest();
@@ -176,6 +178,14 @@ protected void serviceInit(Configuration conf) throws
Exception {
// Init startTime to current time. If all goes well, it will be reset after
// first attempt to contact RM.
retrystartTime = System.currentTimeMillis();
+scheduleStats.numMaxMaps = conf.getInt(MRJobConfig.MR_MAP_NUM_MAX,
Integer.MAX_VALUE);
+if (scheduleStats.numMaxMaps = 0) {
+ scheduleStats.numMaxMaps = Integer.MAX_VALUE;
+}
+scheduleStats.numMaxReduces = conf.getInt(MRJobConfig.MR_REDUCE_NUM_MAX,
Integer.MAX_VALUE);
+if (scheduleStats.numMaxReduces = 0) {
+ scheduleStats.numMaxReduces = Integer.MAX_VALUE;
+}
}
@Override
@@ -216,6 +226,7 @@ public void run() {
@Override
protected synchronized void heartbeat() throws Exception {
+scheduleMaps();
scheduleStats.updateAndLogIfChanged(Before Scheduling: );
ListContainer allocatedContainers = getResources();
if (allocatedContainers.size() 0) {
@@ -233,12 +244,13 @@ protected synchronized void heartbeat() throws Exception {
if (recalculateReduceSchedule) {
preemptReducesIfNeeded();
scheduleReduces(
- getJob().getTotalMaps(), completedMaps,
- scheduledRequests.maps.size(), scheduledRequests.reduces.size(),
- assignedRequests.maps.size(), assignedRequests.reduces.size(),
- mapResourceReqt, reduceResourceReqt,
- pendingReduces.size(),
- maxReduceRampupLimit, reduceSlowStart);
+ getJob().getTotalMaps(), completedMaps,
+ scheduledRequests.maps.size() + pendingMaps.size(),
+ scheduledRequests.reduces.size(),
+ assignedRequests.maps.size(), assignedRequests.reduces.size(),
+ mapResourceReqt, reduceResourceReqt,
+ pendingReduces.size(),
+ maxReduceRampupLimit, reduceSlowStart);
recalculateReduceSchedule = false;
}
@@ -313,7 +325,14 @@ protected synchronized void
handleEvent(ContainerAllocatorEvent event) {
//set the rounded off memory
reqEvent.getCapability().setMemory(mapResourceReqt.getMemory());
reqEvent.getCapability().setVirtualCores(mapResourceReqt.getVirtualCores());
-scheduledRequests.addMap(reqEvent);//maps are immediately scheduled
+//scheduledRequests.addMap(reqEvent);//maps are immediately scheduled
+//将map请求添加到pending队列，然后进行调度
+if (reqEvent.getEarlierAttemptFailed()) {
+ pendingMaps.addFirst(reqEvent);
+} else {
+ pendingMaps.add(reqEvent);
+}
+scheduleMaps();
} else {
if (reduceResourceReqt.equals(Resources.none())) {
reduceResourceReqt = reqEvent.getCapability();
@@ -375,6 +394,16 @@ protected synchronized void
handleEvent(ContainerAllocatorEvent event) {
}
}
+ private void scheduleMaps() {
+LOG.info(scheduling maps from pending queue);
+//more map to be scheduled
+int num = scheduleStats.numMaxMaps -
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6176:
Fix Version/s: 2.4.1
Target Version/s: 2.5.2, 2.5.1, 2.4.1, 2.5.0 (was: 2.5.0, 2.4.1, 2.5.1,
2.5.2)
Release Note:
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
index 7c18f06..6f69168 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
@@ -111,16 +111,18 @@
completed - request corresponding to which container has completed
Lifecycle of map
- scheduled-assigned-completed
-
+ pending-scheduled-assigned-completed
+
Lifecycle of reduce
pending-scheduled-assigned-completed
-
- Maps are scheduled as soon as their requests are received. Reduces are
- added to the pending and are ramped up (added to scheduled) based
- on completed maps and current availability in the cluster.
+
+ Maps are added to the pending are scheduled when the assigned maps plus
scheduled map less than max number of map. Reduces are
+ added to the pending and are ramped up (added to scheduled) based
+ on completed maps and current availability in the cluster as well as the max
number of reduce.
*/
-
+ //maps which are not yet scheduled
+ private final LinkedListContainerRequestEvent pendingMaps =
+new LinkedListContainerRequestEvent();
//reduces which are not yet scheduled
private final LinkedListContainerRequest pendingReduces =
new LinkedListContainerRequest();
@@ -176,6 +178,14 @@ protected void serviceInit(Configuration conf) throws
Exception {
// Init startTime to current time. If all goes well, it will be reset after
// first attempt to contact RM.
retrystartTime = System.currentTimeMillis();
+scheduleStats.numMaxMaps = conf.getInt(MRJobConfig.MR_MAP_NUM_MAX,
Integer.MAX_VALUE);
+if (scheduleStats.numMaxMaps = 0) {
+ scheduleStats.numMaxMaps = Integer.MAX_VALUE;
+}
+scheduleStats.numMaxReduces = conf.getInt(MRJobConfig.MR_REDUCE_NUM_MAX,
Integer.MAX_VALUE);
+if (scheduleStats.numMaxReduces = 0) {
+ scheduleStats.numMaxReduces = Integer.MAX_VALUE;
+}
}
@Override
@@ -216,6 +226,7 @@ public void run() {
@Override
protected synchronized void heartbeat() throws Exception {
+scheduleMaps();
scheduleStats.updateAndLogIfChanged(Before Scheduling: );
ListContainer allocatedContainers = getResources();
if (allocatedContainers.size() 0) {
@@ -233,12 +244,13 @@ protected synchronized void heartbeat() throws Exception {
if (recalculateReduceSchedule) {
preemptReducesIfNeeded();
scheduleReduces(
- getJob().getTotalMaps(), completedMaps,
- scheduledRequests.maps.size(), scheduledRequests.reduces.size(),
- assignedRequests.maps.size(), assignedRequests.reduces.size(),
- mapResourceReqt, reduceResourceReqt,
- pendingReduces.size(),
- maxReduceRampupLimit, reduceSlowStart);
+ getJob().getTotalMaps(), completedMaps,
+ scheduledRequests.maps.size() + pendingMaps.size(),
+ scheduledRequests.reduces.size(),
+ assignedRequests.maps.size(), assignedRequests.reduces.size(),
+ mapResourceReqt, reduceResourceReqt,
+ pendingReduces.size(),
+ maxReduceRampupLimit, reduceSlowStart);
recalculateReduceSchedule = false;
}
@@ -313,7 +325,14 @@ protected synchronized void
handleEvent(ContainerAllocatorEvent event) {
//set the rounded off memory
reqEvent.getCapability().setMemory(mapResourceReqt.getMemory());
reqEvent.getCapability().setVirtualCores(mapResourceReqt.getVirtualCores());
-scheduledRequests.addMap(reqEvent);//maps are immediately scheduled
+//scheduledRequests.addMap(reqEvent);//maps are immediately scheduled
+//将map请求添加到pending队列，然后进行调度
+if (reqEvent.getEarlierAttemptFailed()) {
+ pendingMaps.addFirst(reqEvent);
+} else {
+ pendingMaps.add(reqEvent);
+}
+scheduleMaps();
} else {
if (reduceResourceReqt.equals(Resources.none())) {
reduceResourceReqt = reqEvent.getCapability();
@@ -375,6 +394,16 @@ protected synchronized void
handleEvent(ContainerAllocatorEvent event) {
}
}
+ private void scheduleMaps() {
+LOG.info(scheduling maps from
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Target Version/s: 2.5.2, 2.5.1, 2.4.1, 2.5.0 (was: 2.5.0, 2.4.1, 2.5.1, 2.5.2) Status: Open (was: Patch Available) Users should limit the number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.2, 2.5.1, 2.4.1, 2.5.0 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 As MapReduce is batch framework of calculation, so people may want to run application A as well as application B . A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: MAPREDUCE-6176.patch Users should limit the number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is batch framework of calculation, so people may want to run application A as well as application B . A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Description: As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R (was: As MapReduce is batch framework of calculation, so people may want to run application A as well as application B . A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R) Target Version/s: 2.5.2, 2.5.1, 2.4.1, 2.5.0 (was: 2.5.0, 2.4.1, 2.5.1, 2.5.2) Users should limit the number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: MAPREDUCE-6176.patch Users should limit the number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) Users should limit the number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Attachment: (was: MAPREDUCE-6176.patch) Users should limit the number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the number of map task or reduce number an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Summary: To limit the number of map task or reduce number an application (was: Users should limit the number of an application) To limit the number of map task or reduce number an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Summary: To limit the map task number or reduce number of an application (was: To limit the number of map task or reduce number an application) To limit the map task number or reduce number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Description: As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R (was: As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, but a limit resource should be added on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R) To limit the map task number or reduce number of an application --- Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6176) To limit the map task number or reduce task number of an application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6176: Summary: To limit the map task number or reduce task number of an application (was: To limit the map task number or reduce number of an application) To limit the map task number or reduce task number of an application Key: MAPREDUCE-6176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6176 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, mrv2 Affects Versions: 2.5.0, 2.4.1, 2.5.1, 2.5.2 Reporter: Yang Hao Assignee: Yang Hao Labels: patch Fix For: 2.4.1 Attachments: MAPREDUCE-6176.patch As MapReduce is a batch framework of calculation, so people may want to run application A as well as application B 、C, and a limit resource be put on A. A good way to do so is that we can limit the number of application's map task or reduce task. If we set mapreduce.map.num.max as M, then the map task number will not exceed M. At the same time, if we set mapreduce.map.num.max as R, then the reduce task number will not exceed R -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2398) MRBench: setting the baseDir parameter has no effect
[
https://issues.apache.org/jira/browse/MAPREDUCE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-2398:
Tags: MRBench
Fix Version/s: (was: 0.22.1)
2.5.0
1.3.0
Assignee: Yang Hao
Target Version/s: 2.3.0, 2.5.0
Affects Version/s: 2.5.0
2.4.0
2.3.0
Status: In Progress (was: Patch Available)
This problem still exists in the version Hadoop-2.3.0, so please fix it in the
java org.apache.hadoop.mapred.MRBench.Sometimes we don't have the permission
to operate all the HDFS file, so we have to change the BASE_DIR , and the
INPUT_DIR and OUTPUT_DIR should be updated when BASE_DIR is set. Only two extra
lines be put into the java source code can solve the bug
MRBench: setting the baseDir parameter has no effect
Key: MAPREDUCE-2398
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2398
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Affects Versions: 2.3.0, 0.22.0, 0.20.2, 2.4.0, 2.5.0
Reporter: Michael Noll
Assignee: Yang Hao
Priority: Minor
Fix For: 1.3.0, 2.5.0
Attachments: MAPREDUCE-2398-trunk.patch, MAPREDUCE-2398_0.20.2.patch,
MAPREDUCE-2398_v2-0.20.203.0.patch, MAPREDUCE-2398_v2-trunk.patch
The optional {{-baseDir}} parameter lets user specify the base DFS path for
output/input of MRBench.
However, the two private variables {{INPUT_DIR}} and {{OUTPUT_DIR}}
(MRBench.java) are not updated in the case that the default value of
{{-baseDir}} is actually overwritten by the user. Hence any input and output
is always written to the default locations ({{/benchmarks/MRBench/...}}),
even though the user-supplied location for {{-baseDir}} is created (and
eventually deleted again) on HDFS.
The bug affects at least Hadoop 0.20.2 and the current trunk (r1082703) as of
March 21, 2011.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-2398) MRBench: setting the baseDir parameter has no effect
[
https://issues.apache.org/jira/browse/MAPREDUCE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-2398:
Target Version/s: 2.3.0, 2.4.0, 2.5.0 (was: 2.3.0, 2.5.0)
MRBench: setting the baseDir parameter has no effect
Key: MAPREDUCE-2398
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2398
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Affects Versions: 0.20.2, 0.22.0, 2.3.0, 2.4.0, 2.5.0
Reporter: Michael Noll
Assignee: Yang Hao
Priority: Minor
Fix For: 1.3.0, 2.5.0
Attachments: MAPREDUCE-2398-trunk.patch, MAPREDUCE-2398_0.20.2.patch,
MAPREDUCE-2398_v2-0.20.203.0.patch, MAPREDUCE-2398_v2-trunk.patch
The optional {{-baseDir}} parameter lets user specify the base DFS path for
output/input of MRBench.
However, the two private variables {{INPUT_DIR}} and {{OUTPUT_DIR}}
(MRBench.java) are not updated in the case that the default value of
{{-baseDir}} is actually overwritten by the user. Hence any input and output
is always written to the default locations ({{/benchmarks/MRBench/...}}),
even though the user-supplied location for {{-baseDir}} is created (and
eventually deleted again) on HDFS.
The bug affects at least Hadoop 0.20.2 and the current trunk (r1082703) as of
March 21, 2011.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
