[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes
[
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572271#comment-13572271
]
Hadoop QA commented on MAPREDUCE-4905:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568184/MAPREDUCE-4905-trunk-b.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3309//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3309//console
This message is automatically generated.
test org.apache.hadoop.mapred.pipes
---
Key: MAPREDUCE-4905
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
Project: Hadoop Map/Reduce
Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
Attachments: MAPREDUCE-4905-trunk-a.patch,
MAPREDUCE-4905-trunk-b.patch, MAPREDUCE-4905-trunk.patch
tests for org.apache.hadoop.mapred.pipes
patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4953) HadoopPipes misuses fprintf
[
https://issues.apache.org/jira/browse/MAPREDUCE-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572369#comment-13572369
]
Hudson commented on MAPREDUCE-4953:
---
Integrated in Hadoop-Hdfs-0.23-Build #517 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/517/])
MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via tgraves)
(Revision 1442609)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1442609
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
HadoopPipes misuses fprintf
---
Key: MAPREDUCE-4953
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4953
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Fix For: 2.0.3-alpha, 0.23.7
Attachments: mapreduce-4953.txt
{code}
[exec]
/mnt/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:130:58:
warning: format not a string literal and no format arguments
[-Wformat-security]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572378#comment-13572378 ] Avner BenHanoch commented on MAPREDUCE-4977: Hi Alejandro, This document looks really nice! 2 minor comments: 1. In one place I saw *ShuffleConsumerProvider* instead of *ShuffleConsumerPlugin* 2. Can you please change: using a different application protocol other than HTTP to something like: using a different application protocol other than HTTP such as RDMA Thanks, Avner Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572436#comment-13572436 ] Arun A K commented on MAPREDUCE-4974: - Kindly advice if the optimization is worth. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.20.204.0, 0.24.0 Attachments: MAPREDUCE-4974.1.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572521#comment-13572521 ] Hudson commented on MAPREDUCE-4905: --- Integrated in Hadoop-trunk-Commit #3331 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3331/]) MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes (Aleksey Gorshkov via bobby) (Revision 1443027) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1443027 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/CommonStub.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/PipeApplicationRunnableStub.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/PipeApplicationStub.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/PipeReducerStub.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipesNonJavaInputFormat.java test org.apache.hadoop.mapred.pipes --- Key: MAPREDUCE-4905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Aleksey Gorshkov Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk-b.patch, MAPREDUCE-4905-trunk.patch tests for org.apache.hadoop.mapred.pipes patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4905: --- Resolution: Fixed Fix Version/s: (was: 0.23.6) 0.23.7 Assignee: Aleksey Gorshkov Status: Resolved (was: Patch Available) Thanks Aleksey, I put this into trunk, branch-2, and branch-0.23. test org.apache.hadoop.mapred.pipes --- Key: MAPREDUCE-4905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 2.0.3-alpha, 0.23.7 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk-b.patch, MAPREDUCE-4905-trunk.patch tests for org.apache.hadoop.mapred.pipes patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[
https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572534#comment-13572534
]
Surenkumar Nihalani commented on MAPREDUCE-4974:
One advantage we also might not be able to quantify easily would be in garbage
collection.
I was looking through implementation of {{LineRecordReader.nextKeyValue}}, in
the end of the parsing, we set key and value to null. In the next call, we will
have them as null and if the while condition of {{getFilePosition = end}}
evaluates to {{true}}, then, we'll hit NPE because in.readLine(Text t..) does
t.clear() first. Would there be any case in which the condition would evaluate
to true even when we are done?
I know this probably won't happen because user will wrap this in a
{{while(nextKeyValue())}}, I just wanted to be sure that we won't hit NPEs
after this change, even for buggy programs.
Optimising the LineRecordReader initialize() method
---
Key: MAPREDUCE-4974
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, mrv2, performance
Affects Versions: 2.0.2-alpha, 0.23.5
Environment: Hadoop Linux
Reporter: Arun A K
Assignee: Gelesh
Labels: patch, performance
Fix For: 0.20.204.0, 0.24.0
Attachments: MAPREDUCE-4974.1.patch
Original Estimate: 1h
Remaining Estimate: 1h
I found there is a a scope of optimizing the code, over initialize() if we
have compressionCodecs codec instantiated only if its a compressed input.
Mean while Gelesh George Omathil, added if we could avoid the null check of
key value. This would time save, since for every next key value generation,
null check is done. The intention being to instantiate only once and avoid
NPE as well. Hope both could be met if initialize key value over
initialize() method. We both have worked on it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572553#comment-13572553 ] Steve Loughran commented on MAPREDUCE-4980: --- Are you trying to run all MR jobs in parallel? Because I worry about conditions where the tests are making some assumption about unique access to the filesystem or mini clusters. The MiniMR and MiniDFS clusters do come up on new ports -but I'm not sure about the local filesystem test directories Parallel test execution of hadoop-mapreduce-client-core --- Key: MAPREDUCE-4980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-4980.1.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4822:
--
Status: Open (was: Patch Available)
I see there's another unnecessary conversion category that we've missed, and
that's the TaskIDs and TaskAttemptIDs. They look like the following:
{code}
public TaskAttemptID getAttemptId() {
return TaskAttemptID.forName(attemptId.toString());
}
..
/** Get task id */
public TaskID getTaskId() { return TaskID.forName(taskid.toString()); }
/** Get successful task attempt id */
public TaskAttemptID getSuccessfulTaskAttemptId() {
if(successfulAttemptId != null)
{
return TaskAttemptID.forName(successfulAttemptId.toString());
}
return null;
}
{code}
Sorry I didn't catch this previously.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572622#comment-13572622
]
Chu Tong commented on MAPREDUCE-4822:
-
When I look at the code the TaskAttemptID.forName method does not look like a
simple conversion. Therefore, I think it is safer to leave what it is right now.
public static TaskAttemptID forName(String str
) throws IllegalArgumentException {
if(str == null)
return null;
String exceptionMsg = null;
try {
String[] parts = str.split(Character.toString(SEPARATOR));
if(parts.length == 6) {
if(parts[0].equals(ATTEMPT)) {
String type = parts[3];
TaskType t = TaskID.getTaskType(type.charAt(0));
if(t != null) {
return new org.apache.hadoop.mapred.TaskAttemptID
(parts[1],
Integer.parseInt(parts[2]),
t, Integer.parseInt(parts[4]),
Integer.parseInt(parts[5]));
} else
exceptionMsg = Bad TaskType identifier. TaskAttemptId string :
+ str + is not properly formed.;
}
}
} catch (Exception ex) {
//fall below
}
if (exceptionMsg == null) {
exceptionMsg = TaskAttemptId string : + str
+ is not properly formed;
}
throw new IllegalArgumentException(exceptionMsg);
}
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572624#comment-13572624
]
Chu Tong commented on MAPREDUCE-4822:
-
Sorry, I did not put the code in codeblock style.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572627#comment-13572627 ] Gelesh commented on MAPREDUCE-4974: --- [~snihalani], .. while condition of getFilePosition = end evaluates to true, then, we'll hit NPE .. The Text object value, which is pased to readLine, would not be null, since that is taken care at initialize method, which is called prior to nextKeyValue(). While(nextKeyValue()) loop would end at once, the newSize (the size of newly fetched value equals zero. Here Key And Value , are set to null. But they aren't referred any more after While(nextKeyValue()) loop, and so NPE is not likely to occur. Please verify, and kindly correct me if we have gone wrong, some where. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.20.204.0, 0.24.0 Attachments: MAPREDUCE-4974.1.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572645#comment-13572645
]
Jason Lowe commented on MAPREDUCE-4822:
---
That code simply parses a string into a TaskAttemptID, the very thing we're
trying to avoid doing unnecessarily when we already have the TaskAttemptID. Is
there something specific about that conversion you're concerned is necessary
when passing an existing TaskAttemptID as a string?
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
Plamen Jeliazkov created MAPREDUCE-4981: --- Summary: WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4981: Description: https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path was: https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4981: Attachment: MAPREDUCE-4981.patch WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572691#comment-13572691 ] Plamen Jeliazkov commented on MAPREDUCE-4981: - Uploaded first patch to add existing examples to ExampleDriver. WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4981: Status: Patch Available (was: Open) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4982) AM hung with one pending map task
Jason Lowe created MAPREDUCE-4982: - Summary: AM hung with one pending map task Key: MAPREDUCE-4982 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4982 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Jason Lowe Saw a job that hung with one pending map task that never ran. The task was in the SCHEDULED state with a single attempt that was in the UNASSIGNED state. The AM looked like it was waiting for a container from the RM, but the RM was never granting it the one container it needed. I suspect the AM botched the container request bookkeeping somehow. More details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4977: -- Attachment: MAPREDUCE-4877.patch Thansk Avner. Attaching updated patch integrating comments from Avner and Asokan. This patch is only documentation. Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[
https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572721#comment-13572721
]
Hadoop QA commented on MAPREDUCE-4981:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568272/MAPREDUCE-4981.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-examples.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3310//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3310//console
This message is automatically generated.
WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
---
Key: MAPREDUCE-4981
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Plamen Jeliazkov
Assignee: Plamen Jeliazkov
Priority: Minor
Fix For: 3.0.0, 2.0.3-alpha
Attachments: MAPREDUCE-4981.patch
https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but
they were never added to the ExamplesDriver.
This JIRA proposes to add them to the ExamplesDriver. I have ran them myself
and can confirm the examples still work as intended.
As a workaround for now, people can still run them by:
bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output
dir path
bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path
output dir path
bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir
path output dir path
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572729#comment-13572729 ] Aaron T. Myers commented on MAPREDUCE-4977: --- +1, the patch looks good to me. Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572730#comment-13572730 ] Plamen Jeliazkov commented on MAPREDUCE-4981: - https://reviews.apache.org/r/1091/ already introduced the unit tests needed. WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4976) Use the new StringUtils methods added by HADOOP-9252
[ https://issues.apache.org/jira/browse/MAPREDUCE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-4976: -- Description: HADOOP-9252 slightly changed the format of some StringUtils outputs. Some methods were deprecated by HADOOP-9252. The use of them should be replaced with the new methods. (was: HADOOP-9252 slightly changes the format of some StringUtils outputs. It may cause test failures. Also, some methods were deprecated by HADOOP-9252. The use of them should be replaced with the new methods.) Issue Type: Improvement (was: Bug) Summary: Use the new StringUtils methods added by HADOOP-9252 (was: Fix test failure for HADOOP-9252) A recent Jerkins build [build #3309|https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3309/] shows that HADOOP-9252 does not cause test failure in MapReduce. (Revised summary and description.) Use the new StringUtils methods added by HADOOP-9252 Key: MAPREDUCE-4976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4976 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor HADOOP-9252 slightly changed the format of some StringUtils outputs. Some methods were deprecated by HADOOP-9252. The use of them should be replaced with the new methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2669) Some new examples and test cases for them.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov reassigned MAPREDUCE-2669: --- Assignee: Plamen Jeliazkov Some new examples and test cases for them. -- Key: MAPREDUCE-2669 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2669 Project: Hadoop Map/Reduce Issue Type: Test Components: examples Affects Versions: 0.22.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0 Attachments: MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, MAPREDUCE-2669.patch, mapreduce-new-examples-0.22.patch Original Estimate: 48h Remaining Estimate: 48h Looking to add some more examples such as Mean, Median, and Standard Deviation to the examples. I have some generic JUnit testcases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4981: Description: https://issues.apache.org/jira/browse/MAPREDUCE-2669 introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path was: https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://issues.apache.org/jira/browse/MAPREDUCE-2669 introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
[ https://issues.apache.org/jira/browse/MAPREDUCE-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4981: Description: https://issues.apache.org/jira/browse/MAPREDUCE-2669 introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path Post-patch, people will be able to run them by: bin/hadoop jar /HADOOP_PATH/share/lib/mapreduce-examples.jar wordmean|wordmedian|wordstandarddeviation input file/dir path output dir path Just like they do for running the wordcount example. was: https://issues.apache.org/jira/browse/MAPREDUCE-2669 introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver --- Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4981.patch https://issues.apache.org/jira/browse/MAPREDUCE-2669 introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. As a workaround for now, people can still run them by: bin/hadoop org.apache.hadoop.examples.WordMean input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordMedian input file/dir path output dir path bin/hadoop org.apache.hadoop.examples.WordStandardDeviation input file/dir path output dir path Post-patch, people will be able to run them by: bin/hadoop jar /HADOOP_PATH/share/lib/mapreduce-examples.jar wordmean|wordmedian|wordstandarddeviation input file/dir path output dir path Just like they do for running the wordcount example. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4977: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572749#comment-13572749
]
Chu Tong commented on MAPREDUCE-4822:
-
My primary concern is the error checking to throw the exception at the end of
the function and I do not know if we want to get ride of that.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572750#comment-13572750
]
Hadoop QA commented on MAPREDUCE-4977:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568279/MAPREDUCE-4877.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3311//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3311//console
This message is automatically generated.
Documentation for pluggable shuffle and pluggable sort
--
Key: MAPREDUCE-4977
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.0.3-alpha
Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch
Add documentation with basic information on pluggable shuffle and sort.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572763#comment-13572763 ] Hudson commented on MAPREDUCE-4977: --- Integrated in Hadoop-trunk-Commit #3334 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3334/]) MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort. (tucu) (Revision 1443168) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1443168 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4982) AM hung with one pending map task
[
https://issues.apache.org/jira/browse/MAPREDUCE-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572776#comment-13572776
]
Jason Lowe commented on MAPREDUCE-4982:
---
Note that this job had many map attempt failures, and a number of nodes had
been blacklisted by the AM as a result. At one point in the log I saw this
message which was a bit troubling:
{noformat}
2013-02-03 16:30:32,164 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not map
allocated container to a valid request. Releasing allocated container
Container: [ContainerId: container_1359150825713_856434_01_003359, NodeId: xx,
NodeHttpAddress: xx, Resource: memory: 4608, Priority: {Priority: 5}, State:
NEW, Token: ContainerToken { kind: ContainerToken, service: xx }, Status:
container_id {, app_attempt_id {, application_id {, id: 856434,
cluster_timestamp: 1359150825713, }, attemptId: 1, }, id: 3359, }, state:
C_NEW, ]
{noformat}
I suspect the AM couldn't associate it with an outstanding map task and lost
the container, and that container is effectively the one needed to complete the
final map task and therefore the job.
Note that the priority of the missing container is for a failed map. I'm
wondering if a failed map somehow stole a normal priority request, and when the
failed map priority request finally came in there were no more failed attempts
to associate with it and the container was dropped.
AM hung with one pending map task
-
Key: MAPREDUCE-4982
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4982
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Jason Lowe
Saw a job that hung with one pending map task that never ran. The task was
in the SCHEDULED state with a single attempt that was in the UNASSIGNED
state. The AM looked like it was waiting for a container from the RM, but
the RM was never granting it the one container it needed.
I suspect the AM botched the container request bookkeeping somehow. More
details to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572801#comment-13572801
]
Jason Lowe commented on MAPREDUCE-4822:
---
The intent of TaskAttemptID.forName() is to be the inverse of its toString().
In these cases we have a valid TaskAttemptID, so the IllegalArgumentException
should never be thrown in this context unless forName() or toString() is
broken. And I can't imagine why someone would expect and rely on that behavior
if it somehow were throwing it. We have an ID, someone is asking for it, why
convert it to a string and back before returning it?
To sum up, {{return TaskAttemptID.forName(taskAttempt.toString())}} should just
be {{return taskAttempt}}. If you're not comfortable making the change, that's
fine let me know. We can either file a followup JIRA or I can post the changes
to the patch.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572867#comment-13572867
]
Chu Tong commented on MAPREDUCE-4822:
-
Ok, I think your point is more solid. I made the changes as you instructed.
Thank you.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572894#comment-13572894
]
Hadoop QA commented on MAPREDUCE-4822:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568300/MAPREDUCE-4822.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3312//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3312//console
This message is automatically generated.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4637) Killing an unassigned task attempt causes the job to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4637: - Fix Version/s: 0.23.7 Killing an unassigned task attempt causes the job to fail - Key: MAPREDUCE-4637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Mayank Bansal Fix For: 2.0.3-alpha, 0.23.7 Attachments: MAPREDUCE-4637-trunk.patch, MAPREDUCE-4637-trunk-v2.patch, MAPREDUCE-4637-trunk-v3.patch, MAPREDUCE-4637-trunk-v4.patch, MAPREDUCE-4637-trunk-v5.patch, MAPREDUCE-4637-trunk-v6.patch, MapReduce.png Attempting to kill a task attempt that has been scheduled but is not running causes an invalid state transition and the AM to stop with an error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572934#comment-13572934
]
Jason Lowe commented on MAPREDUCE-4822:
---
+1, lgtm. There's one minor nit where we're now needlessly checking for null:
{code}
if(successfulAttemptId != null)
{
return successfulAttemptId;
}
return null;
{code}
but I'll just fix that as part of the commit.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572953#comment-13572953
]
Chu Tong commented on MAPREDUCE-4822:
-
right ..., Thanks Jason!
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
Chris Nauroth created MAPREDUCE-4983: Summary: multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[
https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572963#comment-13572963
]
Chris Nauroth commented on MAPREDUCE-4983:
--
The impacted tests are: {{TestMRApps}}, {{TestMapProgress}}, {{TestMRJobs}},
and {{TestUberAM}}.
multiple MapReduce tests fail on Windows due to platform-specific assumptions
in test code
--
Key: MAPREDUCE-4983
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Multiple MapReduce tests have code that makes platform-specific assumptions
which do not hold true on Windows. This includes assumptions about file path
manipulation, the path separator used between classpath elements, environment
variable syntax, and order of files returned from a directory listing of the
local file system.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572965#comment-13572965
]
Hudson commented on MAPREDUCE-4822:
---
Integrated in Hadoop-trunk-Commit #3335 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3335/])
MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by
Chu Tong (Revision 1443257)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1443257
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
Sandy Ryza created MAPREDUCE-4984: - Summary: AM leaves unwanted node/rack requests after container has been assigned Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4822:
--
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
0.23.7
2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Chu! I committed this to trunk, branch-2, and branch-0.23.
Note that for future JIRAs, the Fix Version field should only be set when the
fix is finally committed. The Target Version field should be used when
indicating what version(s) a patch is targeting before it is committed.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 2.0.3-alpha, 0.23.7
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[
https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4983:
-
Attachment: MAPREDUCE-4983-branch-trunk-win.1.patch
The attached patch changes the test code so that it's not platform-specific. I
tested the patch on Mac and Windows. This patch is intended for
branch-trunk-win (not trunk).
Here is a summary of the kinds of changes that are in this patch:
{code}
private static void delete(File dir) throws IOException {
-Path p = new Path(file://+dir.getAbsolutePath());
Configuration conf = new Configuration();
-FileSystem fs = p.getFileSystem(conf);
+FileSystem fs = FileSystem.getLocal(conf);
+Path p = fs.makeQualified(new Path(dir.getAbsolutePath()));
fs.delete(p, true);
}
{code}
The prior code for path construction fails on Windows due to the drive spec and
backslashes. Using {{FileSystem#makeQualified}} against the local file system
works cross-platform.
{code}
-assertTrue(environment.get(CLASSPATH).startsWith($PWD:));
+assertTrue(environment.get(CLASSPATH).startsWith(
+ ApplicationConstants.Environment.PWD.$() + File.pathSeparator));
{code}
On Windows, an environment variable shows up in the classpath as %VAR% instead
of $VAR. On branch-trunk-win, we have already changed
{{ApplicationConstants#Environment#$}} to return the correct thing depending on
platform, so I'm reusing that here.
{code}
- yarnAppClasspath = yarnAppClasspath.replaceAll(,\\s*, :).trim();
+ yarnAppClasspath = yarnAppClasspath.replaceAll(,\\s*,
File.pathSeparator)
+.trim();
{code}
On Windows, classpath entries are separated by ';' instead of ':'. Using
{{File#pathSeparator}} handles this correctly cross-platform.
{code}
-assertSame(MAPREDUCE_JOB_USER_CLASSPATH_FIRST set, but not taking
effect!,
-
env_str.indexOf($PWD:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:$PWD/*),
0);
+String expectedClasspath = StringUtils.join(File.pathSeparator,
+ Arrays.asList(ApplicationConstants.Environment.PWD.$(),
job.jar/job.jar,
+job.jar/classes/, job.jar/lib/*,
+ApplicationConstants.Environment.PWD.$() + /*));
+assertTrue(MAPREDUCE_JOB_USER_CLASSPATH_FIRST set, but not taking
effect!,
+ env_str.startsWith(expectedClasspath));
{code}
This is a combination of the prior issues: handling environment variables and
classpath entry separator in a way that works cross-platform.
{code}
- private static String TEST_ROOT_DIR = new File(System.getProperty(
- test.build.data, /tmp)).getAbsolutePath() + /mapPahseprogress;
+ private static String TEST_ROOT_DIR;
+ static {
+String root = new File(System.getProperty(test.build.data, /tmp))
+ .getAbsolutePath();
+TEST_ROOT_DIR = new Path(root, mapPahseprogress).toString();
+ }
{code}
The old code would generate a path with a mix of backslashes and forward
slashes. Passing through {{Path#toString}} handles this correctly.
{code}
-new Path(mrCluster.getTestWorkDir().getAbsolutePath(),
random-output);
+new Path(/tmp/ + getClass().getSimpleName(), random-output);
{code}
The old code would attempt to use a path on HDFS with a drive spec. HDFS would
reject this, because it considers ':' invalid in a path. See prior discussion
in HDFS-4470, HADOOP-8487, and HDFS-4260 for discussion and justification for
switching to a path of the form /tmp/test name. Note that this does not
change any paths used on the local file system. This only changes paths used
for creating files in HDFS.
{code}
// Check lengths of the files
- Assert.assertEquals(1, localFs.getFileStatus(files[1]).getLen());
- Assert.assertTrue(localFs.getFileStatus(files[2]).getLen() 1);
+ MapString, Path filesMap = pathsToMap(files);
+ Assert.assertTrue(filesMap.containsKey(distributed.first.symlink));
+ Assert.assertEquals(1, localFs.getFileStatus(
+filesMap.get(distributed.first.symlink)).getLen());
+ Assert.assertTrue(filesMap.containsKey(distributed.second.jar));
+ Assert.assertTrue(localFs.getFileStatus(
+filesMap.get(distributed.second.jar)).getLen() 1);
{code}
The old code assumed that the directory listing would come back in a specific
order. The order can be different on Windows. Additionally, Windows has an
additional jar used to bundle a potentially long classpath into a jar manifest.
The new code creates a mapping based on name and then interrogates the map, so
there is no assumption of order.
With this patch, we still have a failure in
{{TestMRJobs#testDistributedCache}}, but it's a separate issue due to
mishandling of symlinks in the distributed cache. I'll file a separate jira
for that.
multiple MapReduce tests fail on Windows due to platform-specific assumptions
in test code
[jira] [Commented] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
[ https://issues.apache.org/jira/browse/MAPREDUCE-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573006#comment-13573006 ] Siddharth Seth commented on MAPREDUCE-4984: --- Sandy, is this a dupe of MAPREDUCE-4671 ? AM leaves unwanted node/rack requests after container has been assigned --- Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
[ https://issues.apache.org/jira/browse/MAPREDUCE-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573015#comment-13573015 ] Sandy Ryza commented on MAPREDUCE-4984: --- Oops, looks like it is. AM leaves unwanted node/rack requests after container has been assigned --- Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
[ https://issues.apache.org/jira/browse/MAPREDUCE-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-4984. --- Resolution: Duplicate AM leaves unwanted node/rack requests after container has been assigned --- Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4985) TestDFSIO supports compression but usages doesn't reflect
Plamen Jeliazkov created MAPREDUCE-4985: --- Summary: TestDFSIO supports compression but usages doesn't reflect Key: MAPREDUCE-4985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4985 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Trivial Fix For: 3.0.0 https://issues.apache.org/jira/browse/MAPREDUCE-2786 introduced the ability to use a compression codec during TestDFSIO. However, the -compression parameter was never introduced to the usages printout. This is a trivial patch to reveal the parameter to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4985) TestDFSIO supports compression but usages doesn't reflect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4985: Attachment: MAPREDUCE-4985.patch TestDFSIO supports compression but usages doesn't reflect - Key: MAPREDUCE-4985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4985 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4985.patch https://issues.apache.org/jira/browse/MAPREDUCE-2786 introduced the ability to use a compression codec during TestDFSIO. However, the -compression parameter was never introduced to the usages printout. This is a trivial patch to reveal the parameter to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4985) TestDFSIO supports compression but usages doesn't reflect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated MAPREDUCE-4985: Status: Patch Available (was: Open) TestDFSIO supports compression but usages doesn't reflect - Key: MAPREDUCE-4985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4985 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4985.patch https://issues.apache.org/jira/browse/MAPREDUCE-2786 introduced the ability to use a compression codec during TestDFSIO. However, the -compression parameter was never introduced to the usages printout. This is a trivial patch to reveal the parameter to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573022#comment-13573022 ] Surenkumar Nihalani commented on MAPREDUCE-4974: I am referring to a buggy program that calls nextKeyValue even after we return false. I just wanted to be sure that the check in the while loop will guard us from calling in.readLine(). in.readLine when passed a null Text type, we will hit NPE. That's the case I am trying to be safe about. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.20.204.0, 0.24.0 Attachments: MAPREDUCE-4974.1.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573024#comment-13573024
]
Hadoop QA commented on MAPREDUCE-4671:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551323/MAPREDUCE-4671.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3313//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3313//console
This message is automatically generated.
AM does not tell the RM about container requests that are no longer needed
--
Key: MAPREDUCE-4671
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch,
MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch
Say the AM wanted a container at hosts h1, h2, h3. After getting a container
at h1 it should tell RM that it no longer needs containers at h2, h3.
Otherwise on the RM h2, h3 remain valid allocation locations.
The AM RMContainerAllocator does remove these resource requests internally.
When the resource request container count drops to 0 then it drops the
resource request from its tables but forgets to send the 0 sized request to
the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4986) Document the locality effect of preemption timeouts in FairScheduler when mapred.fairscheduler.preemption is off
Gera Shegalov created MAPREDUCE-4986: Summary: Document the locality effect of preemption timeouts in FairScheduler when mapred.fairscheduler.preemption is off Key: MAPREDUCE-4986 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4986 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share, documentation Affects Versions: 1.1.1 Reporter: Gera Shegalov fair_scheduler.xml does not mention the delay scheduling algorithm. Furthermore, it does not explain that in getAllowedLocalityLevel the preemption timeouts are used to disable the locality delay scheduling for the starved pool when mapred.fairscheduler.preemption=false (default). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4986) Document the locality effect of preemption timeouts in FairScheduler when mapred.fairscheduler.preemption is off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4986: - Attachment: MAPREDUCE-4986-branch-1.patch Document the locality effect of preemption timeouts in FairScheduler when mapred.fairscheduler.preemption is off Key: MAPREDUCE-4986 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4986 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share, documentation Affects Versions: 1.1.1 Reporter: Gera Shegalov Attachments: MAPREDUCE-4986-branch-1.patch fair_scheduler.xml does not mention the delay scheduling algorithm. Furthermore, it does not explain that in getAllowedLocalityLevel the preemption timeouts are used to disable the locality delay scheduling for the starved pool when mapred.fairscheduler.preemption=false (default). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4985) TestDFSIO supports compression but usages doesn't reflect
[
https://issues.apache.org/jira/browse/MAPREDUCE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573061#comment-13573061
]
Hadoop QA commented on MAPREDUCE-4985:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568330/MAPREDUCE-4985.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3314//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3314//console
This message is automatically generated.
TestDFSIO supports compression but usages doesn't reflect
-
Key: MAPREDUCE-4985
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4985
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Plamen Jeliazkov
Assignee: Plamen Jeliazkov
Priority: Trivial
Fix For: 3.0.0
Attachments: MAPREDUCE-4985.patch
https://issues.apache.org/jira/browse/MAPREDUCE-2786 introduced the ability
to use a compression codec during TestDFSIO. However, the -compression
parameter was never introduced to the usages printout.
This is a trivial patch to reveal the parameter to end users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573086#comment-13573086 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4983: bq. With this patch, we still have a failure in TestMRJobs#testDistributedCache, but it's a separate issue due to mishandling of symlinks in the distributed cache. I'll file a separate jira for that. Please do. multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573084#comment-13573084 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4983: Mostly looks good. One comment: - Can you change TestMRJobs to use TEST_ROOT_DIR instead of hardcoding /tmp, but address the mixed slash issue with Path#toString()? multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4643: -- Attachment: MAPREDUCE-4643-branch-1-4.patch Make job-history cleanup-period configurable Key: MAPREDUCE-4643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3 Reporter: Karthik Kambatla Assignee: Sandy Ryza Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573091#comment-13573091 ] Sandy Ryza commented on MAPREDUCE-4643: --- Latest patch adds the default values into mapred-default.xml and takes out the test, which will be covered in MAPREDUCE-4676. Make job-history cleanup-period configurable Key: MAPREDUCE-4643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3 Reporter: Karthik Kambatla Assignee: Sandy Ryza Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573096#comment-13573096
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568349/MAPREDUCE-4643-branch-1-4.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3315//console
This message is automatically generated.
Make job-history cleanup-period configurable
Key: MAPREDUCE-4643
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4643-branch1-2.patch,
MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch,
MAPREDUCE-4643.patch
Job history cleanup should be made configurable. Currently, it is set to 1
month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573113#comment-13573113 ] Bikas Saha commented on MAPREDUCE-4983: --- Clean fixes overall. Please do fix the existing typo mapPahseprogress :P multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[
https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4983:
-
Attachment: MAPREDUCE-4983-branch-trunk-win.2.patch
Thanks, Vinod and Bikas. I'm attaching version 2 of the patch to address your
feedback.
{quote}
Can you change TestMRJobs to use TEST_ROOT_DIR instead of hardcoding /tmp, but
address the mixed slash issue with Path#toString()?
{quote}
Unfortunately, no, we can't use TEST_ROOT_DIR for this. In addition to the
slash issue, TEST_ROOT_DIR will have a drive spec, and HDFS will reject any
path with a ':' as invalid. TEST_ROOT_DIR needs to stay fully qualified, with
drive spec, for some of the local file system setup operations in the test. I
have introduced a new constant, OUTPUT_ROOT_DIR, so at least we don't repeat
/tmp in multiple places.
{quote}
Please do fix the existing typo mapPahseprogress{quote}
I kept looking right past it. :-) Fixed.
multiple MapReduce tests fail on Windows due to platform-specific assumptions
in test code
--
Key: MAPREDUCE-4983
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch,
MAPREDUCE-4983-branch-trunk-win.2.patch
Multiple MapReduce tests have code that makes platform-specific assumptions
which do not hold true on Windows. This includes assumptions about file path
manipulation, the path separator used between classpath elements, environment
variable syntax, and order of files returned from a directory listing of the
local file system.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573159#comment-13573159 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4983: Thought this was a local FS path so was advising against /tmp, but realized otherwise now. Better solution could have been Path itself taking care of mixed slashes? Never mind for this patch though.. Checking it in. multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch, MAPREDUCE-4983-branch-trunk-win.2.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-4983. Resolution: Fixed Hadoop Flags: Reviewed Just committed this to branch-trunk-win. Thanks Chris! multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch, MAPREDUCE-4983-branch-trunk-win.2.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4983: --- Fix Version/s: trunk-win multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch, MAPREDUCE-4983-branch-trunk-win.2.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4987:
-
Description: On Windows, {{TestMRJobs#testDistributedCache}} fails on an
assertion while checking the length of a symlink. It expects to see the length
of the target of the symlink, but Java 6 on Windows always reports that a
symlink has length 0. (was: On Windows, TestMRJobs#testDistributedCache fails
on an assertion while checking the length of a symlink. It expects to see the
length of the target of the symlink, but Java 6 on Windows always reports that
a symlink has length 0.)
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: trunk-win
Reporter: Chris Nauroth
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
Chris Nauroth created MAPREDUCE-4987: Summary: TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks Key: MAPREDUCE-4987 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache, nodemanager Affects Versions: trunk-win Reporter: Chris Nauroth On Windows, TestMRJobs#testDistributedCache fails on an assertion while checking the length of a symlink. It expects to see the length of the target of the symlink, but Java 6 on Windows always reports that a symlink has length 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573191#comment-13573191
]
Chris Nauroth commented on MAPREDUCE-4987:
--
In {{TestMRJobs}}, the last assertion in this code fragment fails:
{code}
// Check lengths of the files
MapString, Path filesMap = pathsToMap(files);
Assert.assertTrue(filesMap.containsKey(distributed.first.symlink));
Assert.assertEquals(1, localFs.getFileStatus(
filesMap.get(distributed.first.symlink)).getLen());
{code}
This is a known issue with Java 6 on Windows. It always reports a symlink as
having length zero instead of the length of the target file. This problem was
fixed on branch-1-win in HADOOP-9061 by detecting if the runtime environment is
Windows + Java 6, and if so, copying files into the symlink location instead of
actually creating a symlink. Applying the same logic to branch-trunk-win will
require different code changes. In YARN, the symlinks for the distributed
cache get generated by the container launch scripts. See
{{ContainerLaunch#WindowsShellScriptBuilder#link}}.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: trunk-win
Reporter: Chris Nauroth
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4983) multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
[ https://issues.apache.org/jira/browse/MAPREDUCE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573192#comment-13573192 ] Chris Nauroth commented on MAPREDUCE-4983: -- Thanks for the commit! I filed MAPREDUCE-4987 for the remaining failure. multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code -- Key: MAPREDUCE-4983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4983 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: MAPREDUCE-4983-branch-trunk-win.1.patch, MAPREDUCE-4983-branch-trunk-win.2.patch Multiple MapReduce tests have code that makes platform-specific assumptions which do not hold true on Windows. This includes assumptions about file path manipulation, the path separator used between classpath elements, environment variable syntax, and order of files returned from a directory listing of the local file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573199#comment-13573199 ] Gelesh commented on MAPREDUCE-4974: --- As [~snihalani] has mentioned, a buggy programs that may call next KeyValue.. condition though being a little hypothetical, but still possible. 1) Inorder to avoid that, shall we have the null assignment of key value in close() method.? 2) Also shall, we have compressionCodecs also assigned as null ? Either me or [~ak.a...@aol.com] would upload a re work on the same. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.20.204.0, 0.24.0 Attachments: MAPREDUCE-4974.1.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573202#comment-13573202 ] Gelesh commented on MAPREDUCE-4974: --- Also, this change has instantiated objects related to compression, only if its a compressed file Inorder to ship the first line, a readLine is called, and this change would not create a new Text, but use the available 'value' for the method call. Hope some body could share their thoughts on this two changes as well. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.20.204.0, 0.24.0 Attachments: MAPREDUCE-4974.1.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4821) Unit Test: TestJobTrackerRestart fails when it is run with ant-1.8.4
[ https://issues.apache.org/jira/browse/MAPREDUCE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573208#comment-13573208 ] Amir Sanjar commented on MAPREDUCE-4821: Hi Matt, is it too late to request the patch to be included in 1.1.2 release, thanks Unit Test: TestJobTrackerRestart fails when it is run with ant-1.8.4 Key: MAPREDUCE-4821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4821 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.3, 1.0.4 Environment: RHEL 6.3 on x86 Reporter: Amir Sanjar Attachments: MAPREDUCE-4821-branch1.patch, MAPREDUCE-4821-release-1.0.3.patch Problem: JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not JUnit4: Solution: Migrate the testcase to JUnit4, including: * Remove extends TestCase * Remove import junit.framework.TestCase; * Add import org.junit.*; * Use appropriate annotations such as @After, @Before, @Test. uploading a patch shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4821) Unit Test: TestJobTrackerRestart fails when it is run with ant-1.8.4
[ https://issues.apache.org/jira/browse/MAPREDUCE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Sanjar updated MAPREDUCE-4821: --- Fix Version/s: 1.1.2 Unit Test: TestJobTrackerRestart fails when it is run with ant-1.8.4 Key: MAPREDUCE-4821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4821 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.3, 1.0.4 Environment: RHEL 6.3 on x86 Reporter: Amir Sanjar Fix For: 1.1.2 Attachments: MAPREDUCE-4821-branch1.patch, MAPREDUCE-4821-release-1.0.3.patch Problem: JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not JUnit4: Solution: Migrate the testcase to JUnit4, including: * Remove extends TestCase * Remove import junit.framework.TestCase; * Add import org.junit.*; * Use appropriate annotations such as @After, @Before, @Test. uploading a patch shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573253#comment-13573253 ] Siddharth Seth commented on MAPREDUCE-4671: --- +1. This has been pending for a long time. A variable name needs a minor fix in TestRMCotnaienrAllocator (lastAsk instead of lastAsk). Making this change as part of the commit. AM does not tell the RM about container requests that are no longer needed -- Key: MAPREDUCE-4671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Bikas Saha Assignee: Bikas Saha Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch, MAPREDUCE-4671.txt Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations. The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4671: -- Attachment: MAPREDUCE-4671.txt patch with the variable name fix. AM does not tell the RM about container requests that are no longer needed -- Key: MAPREDUCE-4671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Bikas Saha Assignee: Bikas Saha Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch, MAPREDUCE-4671.txt Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations. The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4671: -- Resolution: Fixed Fix Version/s: 2.0.4-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks Bikas! AM does not tell the RM about container requests that are no longer needed -- Key: MAPREDUCE-4671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.4-beta Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch, MAPREDUCE-4671.txt Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations. The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573261#comment-13573261 ] Hudson commented on MAPREDUCE-4671: --- Integrated in Hadoop-trunk-Commit #3338 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3338/]) MAPREDUCE-4671. AM does not tell the RM about container requests which are no longer needed. Contributed by Bikas Saha. (Revision 1443323) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1443323 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java AM does not tell the RM about container requests that are no longer needed -- Key: MAPREDUCE-4671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.4-beta Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch, MAPREDUCE-4671.txt Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations. The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4972) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-4972: Attachment: MAPREDUCE-4972-trunk.patch MAPREDUCE-4972-branch-0.23.patch Coverage fixing for org.apache.hadoop.mapreduce.jobhistory --- Key: MAPREDUCE-4972 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4972 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv2 Reporter: Aleksey Gorshkov Attachments: HADOOP-9092-branch-0.23-a.patch, HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2-a.patch, HADOOP-9092-branch-2.patch, HADOOP-9092-trunk-a.patch, HADOOP-9092-trunk.patch, MAPREDUCE-4972-branch-0.23.patch, MAPREDUCE-4972-trunk.patch Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4972) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573287#comment-13573287 ] Aleksey Gorshkov commented on MAPREDUCE-4972: - updated patches MAPREDUCE-4972-trunk.patch for trunk and branch-2 MAPREDUCE-4972-branch-0.23.patch for branch-0.23 Coverage fixing for org.apache.hadoop.mapreduce.jobhistory --- Key: MAPREDUCE-4972 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4972 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv2 Reporter: Aleksey Gorshkov Attachments: HADOOP-9092-branch-0.23-a.patch, HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2-a.patch, HADOOP-9092-branch-2.patch, HADOOP-9092-trunk-a.patch, HADOOP-9092-trunk.patch, MAPREDUCE-4972-branch-0.23.patch, MAPREDUCE-4972-trunk.patch Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
