[jira] [Updated] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5183: -- Fix Version/s: (was: 1.0.4) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign -- Key: MAPREDUCE-5183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, tasktracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Priority: Minor Labels: newbie Attachments: MAPREDUCE-5183.patch This makes looking at progress in the logs unnecessarily confusing. It would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664925#comment-13664925 ] Sandy Ryza commented on MAPREDUCE-5183: --- Thanks for working on this, Niranjan. The patch is looking good. A couple stylistic comments: * The line you changed should be broken up so that it fits within 80 characters per line * There should be a space between the comma and the 0. * There should be 0 or 1 blank lines at the end of the test function, and no blank lines at the end of the for loop in the test. * Tests that run a job on a minicluster generally take over a minute. Would it be possible to either write the test without the whole job/minicluster or put the check inside a relevant test that already exists? Also, and others might disagree, I don't think a test is necessarily required for something cosmetic like this. Also, I removed the fix version, as normally the person who commits the patch sets it at commit time. In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign -- Key: MAPREDUCE-5183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, tasktracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Priority: Minor Labels: newbie Attachments: MAPREDUCE-5183.patch This makes looking at progress in the logs unnecessarily confusing. It would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5237: --- Attachment: MAPREDUCE-5237.2.patch Fix the javac warnings ClusterStatus incompatiblity issues with MR1 Key: MAPREDUCE-5237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch The three functions are: CollectionString getGraylistedTrackerNames() int getGraylistedTrackers() State getJobTrackerState() The're tracker related, such that they are no longer used in MR2. Maybe we should add them and throw UnsupportedOperationException? In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664944#comment-13664944
]
Hadoop QA commented on MAPREDUCE-5237:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584453/MAPREDUCE-5237.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3666//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3666//console
This message is automatically generated.
ClusterStatus incompatiblity issues with MR1
Key: MAPREDUCE-5237
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch
The three functions are:
CollectionString getGraylistedTrackerNames()
int getGraylistedTrackers()
State getJobTrackerState()
The're tracker related, such that they are no longer used in MR2. Maybe we
should add them and throw UnsupportedOperationException?
In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664975#comment-13664975 ] Niranjan Singh commented on MAPREDUCE-5183: --- Thanks, Sandy Ryza for the review comments , this is my first patch so missed out on couple of coding styles in future would see I don't missed out on anything. In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign -- Key: MAPREDUCE-5183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, tasktracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Priority: Minor Labels: newbie Attachments: MAPREDUCE-5183.patch This makes looking at progress in the logs unnecessarily confusing. It would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Attachment: MAPREDUCE-5228-trunk-1.patch Adding the patch. Thanks, Mayank Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce -- Key: MAPREDUCE-5228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5228-trunk-1.patch The enum was used by findCounter(Enum key) to find a specific counter object. Now it seems to be replaced by FileInputFormatCounter and FileOutputFormatCounter. Now the enum seems to be only used internally, but not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Patch Available (was: Open) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce -- Key: MAPREDUCE-5228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5228-trunk-1.patch The enum was used by findCounter(Enum key) to find a specific counter object. Now it seems to be replaced by FileInputFormatCounter and FileOutputFormatCounter. Now the enum seems to be only used internally, but not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5245: --- Attachment: MAPREDUCE-5245.1.patch Add the variables back. Since MAPRED_JOB_MAP_MEMORY_MB_PROPERTY and MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY are changed to new values in M/R 2.x. Therefore, the incompatible changes are introduced. The existing MAPRED_JOB_MAP_MEMORY_MB_PROPERTY and MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY are replaced with MAPREDUCE_JOB_MAP_MEMORY_MB_PROPERTY and MAPREDUCE_JOB_REDUCE_MEMORY_MB_PROPERTY, and MAPRED_JOB_MAP_MEMORY_MB_PROPERTY and MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY are then set to old property names used by M/R 1.x. In addition, set/getMemoryForMapTask() and set/getMemoryForReduceTask() write/read both old and new property names. A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5245.1.patch A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5245: --- Status: Patch Available (was: Open) A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5245.1.patch A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal reassigned MAPREDUCE-5230: Assignee: Mayank Bansal (was: Zhijie Shen) createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5230: - Attachment: MAPREDUCE-5230-trunk-1.patch Adding trivial patch. Thanks, Mayank createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5230: - Status: Patch Available (was: Open) createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[
https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665014#comment-13665014
]
Hadoop QA commented on MAPREDUCE-5230:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584461/MAPREDUCE-5230-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3669//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3669//console
This message is automatically generated.
createFileSplit is removed from NLineInputFormat of mapred
--
Key: MAPREDUCE-5230
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Attachments: MAPREDUCE-5230-trunk-1.patch
createFileSplit is removed from NLineInputFormat of mapred, because it's no
longer used in the new getSplit implementation. However, since function is
protected before, there is still the potential risk that the user defined
format class which extends old NLineInputFormat uses the protected function.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665026#comment-13665026
]
Hadoop QA commented on MAPREDUCE-5228:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584458/MAPREDUCE-5228-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3667//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3667//console
This message is automatically generated.
Enum Counter is removed from FileInputFormat and FileOutputFormat of both
mapred and mapreduce
--
Key: MAPREDUCE-5228
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Attachments: MAPREDUCE-5228-trunk-1.patch
The enum was used by findCounter(Enum key) to find a specific counter object.
Now it seems to be replaced by FileInputFormatCounter and
FileOutputFormatCounter. Now the enum seems to be only used internally, but
not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665032#comment-13665032
]
Hadoop QA commented on MAPREDUCE-5245:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584460/MAPREDUCE-5245.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3668//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3668//console
This message is automatically generated.
A number of public static variables are removed from JobConf
Key: MAPREDUCE-5245
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5245.1.patch
A number of public static variables are removed from JobConf:
boolean DEFAULT_MAPREDUCE_RECOVER_JOB
String MAPREDUCE_RECOVER_JOB
String WORKFLOW_ADJACENCY_PREFIX_PATTERN
String WORKFLOW_ADJACENCY_PREFIX_STRING
String WORKFLOW_ID
String WORKFLOW_NAME
String WORKFLOW_NODE_NAME
String WORKFLOW_TAGS
The workflow related variables are moved to MRJobConfig.
The follwing public static variables becomes default:
String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY
String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY
The variables there are no longer referred internally in 2.x, but they might
be used by users as they were public.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665064#comment-13665064 ] Hudson commented on MAPREDUCE-5233: --- Integrated in Hadoop-Yarn-trunk #218 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/218/]) MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job when compared to 1.x. This breaks 0.23.x users of one API in Job. Contributed by Mayank Bansal. (Revision 1485491) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485491 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/jobcontrol/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/ControlledJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch, MAPREDUCE-5233-trunk-5.patch, MAPREDUCE-5233-trunk-6.patch, MAPREDUCE-5233-trunk-7.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665142#comment-13665142 ] Hudson commented on MAPREDUCE-5233: --- Integrated in Hadoop-Hdfs-trunk #1408 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1408/]) MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job when compared to 1.x. This breaks 0.23.x users of one API in Job. Contributed by Mayank Bansal. (Revision 1485491) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485491 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/jobcontrol/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/ControlledJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch, MAPREDUCE-5233-trunk-5.patch, MAPREDUCE-5233-trunk-6.patch, MAPREDUCE-5233-trunk-7.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665194#comment-13665194 ] Hudson commented on MAPREDUCE-5233: --- Integrated in Hadoop-Mapreduce-trunk #1434 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1434/]) MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job when compared to 1.x. This breaks 0.23.x users of one API in Job. Contributed by Mayank Bansal. (Revision 1485491) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485491 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/jobcontrol/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/ControlledJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch, MAPREDUCE-5233-trunk-5.patch, MAPREDUCE-5233-trunk-6.patch, MAPREDUCE-5233-trunk-7.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5246) Adding application type to submission context
[ https://issues.apache.org/jira/browse/MAPREDUCE-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5246. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed +1. Committed this together with YARN-563 to trunk and branch-2. Adding application type to submission context - Key: MAPREDUCE-5246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5246 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5246-trunk-1.patch, MAPREDUCE-5246-trunk-2.patch, MAPREDUCE-5246-trunk-3-review.patch, MAPREDUCE-5246-trunk-4-review.patch Adding application type to submission context of map reduce YARN-563 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5246) Adding application type to submission context
[ https://issues.apache.org/jira/browse/MAPREDUCE-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665443#comment-13665443 ] Hudson commented on MAPREDUCE-5246: --- Integrated in Hadoop-trunk-Commit #3782 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3782/]) YARN-563. Add the concept of an application-type for each application. Contributed by Mayank Bansal. MAPREDUCE-5246. Specify application-type at the time of job submission after YARN-563. Contributed by Mayank Bansal. (Revision 1485790) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485790 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestApplicatonReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java *
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5263: --- Description: A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK was:A couple of methods and variables have been removed. Target Version/s: 2.0.5-beta filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665618#comment-13665618 ] Mayank Bansal commented on MAPREDUCE-5231: -- +1 looks good to me Thanks, Mayank Constructor of DBInputFormat.DBRecordReader in mapred is changed Key: MAPREDUCE-5231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5231.1.patch The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, if DBInputFormat.DBRecordReader is extended, the incompatibility around the constructor is still there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665677#comment-13665677
]
Ivan Mitic commented on MAPREDUCE-5224:
---
Thanks Xi, you're almost there! A few additional comments below. Once you
address those, +1 from me
1. You'll have to impersonate the MR owner when you're querying for the
systemDirFs (same routine as with defaultFs). Sorry, I missed this when I
initially reviewed the patch.
After:
{code}
if (defaultFs == null) {
defaultFs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
{code}
add the following:
{code}
if (systemDirFs == null) {
systemDirFs = mrOwner.doAs(new
PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return FileSystem.get(sysDir.toUri(), conf);
}});
}
{code}
Once you implement above you should be able to simplify getSystemDir() by
assuming that systemDirFs is different than null. This will allow you to get
rid of the IOException (#2 comment from my initial review).
2. Nit: JobTracker.java: It seems that a tab slipped in:
{code}
if (systemDirFs.exists(restartFile)) {
systemDirFs.delete(tmpRestartFile, false); // delete the tmp file
} else if (systemDirFs.exists(tmpRestartFile)) {
{code}
I also see some invalid indentation:
{code}
// disable recovery if this is a restart
shouldRecover = false;
{code}
Please correct.
3. Please remove try/catch from TestJobTrackerWithNonDefaultFS#tearDown since
it can possibly mask a problem. Instead you can add IOException to the throws
clause of the method:
{code}
public void tearDown() throws IOException {
{code}
4. TestJobTrackerwithNonDefaultFs#testSystemDir: No need for the try/catch
block in the test, please remove. The test will fail if any of its asserts fail.
5. Can you also please change TestJobTrackerWithNonDefaultFS#MAPRED_SYS_DIR to
the following:
{code}
private final String MAPRED_SYS_DIR =
System.getProperty(test.build.data, /tmp) + /mapred/system;
{code}
Guideline is for all local test files to go under test.build.data folder.
6. Nit: TestJobTrackerwithNonDefaultFs: You can use assertTrue instead:
{code}
assertEquals(Check if the system dir exists ,
FileSystem.get(sysDirPathURL, conf).exists(sysDirPath), true);
{code}
Btw, when you’re using assertEquals, you should place the expected value as the
first arg, and the test value as the second arg. For example,
assertEquals(true, fs.get().exists()).
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665679#comment-13665679
]
Xi Fang commented on MAPREDUCE-5224:
Thanks Ivan for your detailed comments. These are of great help!
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665698#comment-13665698 ] Zhijie Shen commented on MAPREDUCE-5230: +1, it's fine that the tests are not included, looks good to me. createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5229) TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665720#comment-13665720 ] Mayank Bansal commented on MAPREDUCE-5229: -- +1 looks good to me. Thanks, Mayank TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce - Key: MAPREDUCE-5229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5229 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5229.1.patch TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce. As FileOutputFormat and FileOutputCommitter may be extend by users, and the extended user classes can configured to use. Therefore, this missing public static variable may cause compatibility problem when the extended classes refer it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5263: --- Attachment: MAPREDUCE-5263.1.patch Added the methods and the variables, annotated them deprecated, and added a test filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5263.1.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5263: --- Status: Patch Available (was: Open) filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5263.1.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
Jian He created MAPREDUCE-5270: -- Summary: Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665723#comment-13665723
]
Mostafa Elhemali commented on MAPREDUCE-5224:
-
Looks good to me. Some small comments mostly about the unit test:
# You don't have to catch(Exception) to call tearDown(), nor to print stack
trace. tearDown() will always be called automatically, and the exception will
be displayed by the test framework. Please remove the try..catch.
# Use assertTrue(cond) instead of assertEquals(cond, true)
# Even though this is a patch for branch-1-win, since it's a new test you
should consider writing it in JUnit 4. See
[http://wiki.apache.org/hadoop/HowToDevelopUnitTests] for guidance.
Other than that, +1 on my side.
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5194:
-
Attachment: M5194-1.patch
Unfortunately, there's no way (AFAICT) to trash a {{HttpURLConnection}}, so
calling {{disconnect()}}- while sufficient to cause the socket close and induce
an error- isn't enough, because each call to {{connect()}} will ignore the
close. So I added a second check after connect to abort if the thread is
stopped.
Heed interrupts during Fetcher shutdown
---
Key: MAPREDUCE-5194
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5194-1.patch, M5194.patch
In the current implementation, {{Fetcher}} instances usually exit gracefully
when the shuffle succeeds. When it fails, threads are interrupted, but may
continue running harmlessly until the JVM shuts down.
However, to generate consistent checkpoints, these threads should exit
cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5194:
-
Status: Patch Available (was: Open)
Heed interrupts during Fetcher shutdown
---
Key: MAPREDUCE-5194
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5194-1.patch, M5194.patch
In the current implementation, {{Fetcher}} instances usually exit gracefully
when the shuffle succeeds. When it fails, threads are interrupted, but may
continue running harmlessly until the JVM shuts down.
However, to generate consistent checkpoints, these threads should exit
cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Attachment: MAPREDUCE-5270.1.patch Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Status: Patch Available (was: Open) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665741#comment-13665741
]
Hadoop QA commented on MAPREDUCE-5270:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584590/MAPREDUCE-5270.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3670//console
This message is automatically generated.
Migrate from using BuilderUtil factory methods to individual record factory
method on MapReduce side
Key: MAPREDUCE-5270
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: MAPREDUCE-5270.1.patch
Migrate the factory method on map reduce side.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665745#comment-13665745 ] Mayank Bansal commented on MAPREDUCE-5237: -- Default state to invalid may break applications, as that was not present before. We should to probably set to running by default. Thanks Mayank ClusterStatus incompatiblity issues with MR1 Key: MAPREDUCE-5237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch The three functions are: CollectionString getGraylistedTrackerNames() int getGraylistedTrackers() State getJobTrackerState() The're tracker related, such that they are no longer used in MR2. Maybe we should add them and throw UnsupportedOperationException? In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665754#comment-13665754
]
Hadoop QA commented on MAPREDUCE-5194:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584589/M5194-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3672//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3672//console
This message is automatically generated.
Heed interrupts during Fetcher shutdown
---
Key: MAPREDUCE-5194
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5194-1.patch, M5194.patch
In the current implementation, {{Fetcher}} instances usually exit gracefully
when the shuffle succeeds. When it fails, threads are interrupted, but may
continue running harmlessly until the JVM shuts down.
However, to generate consistent checkpoints, these threads should exit
cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665755#comment-13665755
]
Hadoop QA commented on MAPREDUCE-5263:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584588/MAPREDUCE-5263.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3671//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3671//console
This message is automatically generated.
filecache.DistributedCache incompatiblity issues with MR1
-
Key: MAPREDUCE-5263
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5263.1.patch
A couple of methods and variables have been removed:
void addLocalArchives(Configuration, String)
void addLocalFiles(Configuration, String)
void createAllSymlink(Configuration, File, File)
FileStatus getFileStatus(Configuration, URI)
long getTimestamp(Configuration, URI)
void setArchiveTimestamps(Configuration, String)
void setFileTimestamps(Configuration, String)
void setLocalArchives(Configuration, String)
void setLocalFiles(Configuration, String)
String CACHE_ARCHIVES
String CACHE_ARCHIVES_SIZES
String CACHE_ARCHIVES_TIMESTAMPS
String CACHE_FILES
String CACHE_FILES_SIZES
String CACHE_FILES_TIMESTAMPS
String CACHE_LOCALARCHIVES
String CACHE_LOCALFILES
String CACHE_SYMLINK
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665794#comment-13665794
]
Mayank Bansal commented on MAPREDUCE-5245:
--
Everything else looks good however we should check mapreduce properties are
also getting set.
{code}
setLong(JobConf.MAPREDUCE_JOB_REDUCE_MEMORY_MB_PROPERTY, mem);
{code}
A number of public static variables are removed from JobConf
Key: MAPREDUCE-5245
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5245.1.patch
A number of public static variables are removed from JobConf:
boolean DEFAULT_MAPREDUCE_RECOVER_JOB
String MAPREDUCE_RECOVER_JOB
String WORKFLOW_ADJACENCY_PREFIX_PATTERN
String WORKFLOW_ADJACENCY_PREFIX_STRING
String WORKFLOW_ID
String WORKFLOW_NAME
String WORKFLOW_NODE_NAME
String WORKFLOW_TAGS
The workflow related variables are moved to MRJobConfig.
The follwing public static variables becomes default:
String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY
String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY
The variables there are no longer referred internally in 2.x, but they might
be used by users as they were public.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5192:
-
Status: Open (was: Patch Available)
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5192:
-
Attachment: M5192-1.patch
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5192:
-
Status: Patch Available (was: Open)
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665831#comment-13665831
]
Hadoop QA commented on MAPREDUCE-5192:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584611/M5192-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3673//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3673//console
This message is automatically generated.
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Attachment: MAPREDUCE-5270.2.patch Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665880#comment-13665880 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5230: Yes. +1 for the straight forward patch. Checking this in. createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5230: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks Mayank! Tx for the quick review, Zhijie! createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665891#comment-13665891 ] Hudson commented on MAPREDUCE-5230: --- Integrated in Hadoop-trunk-Commit #3787 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3787/]) MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary compatibility with mapred in 1.x Contributed by Mayank Bansal. (Revision 1485906) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485906 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/NLineInputFormat.java createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665903#comment-13665903
]
Chris Douglas commented on MAPREDUCE-5192:
--
The findbugs warnings are from renaming {{ShuffleScheduler}} to
{{ShuffleSchedulerImpl}}, but are probably worth addressing
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5192) Separate TCE resolution from fetch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-5192:
-
Status: Open (was: Patch Available)
Separate TCE resolution from fetch
--
Key: MAPREDUCE-5192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Priority: Minor
Attachments: M5192-1.patch, M5192.patch
The {{EventFetcher}} thread grounds task completion events as URIs before
passing them to the {{ShuffleScheduler}}. If the former deferred this to the
scheduler, one could interpret the TCE metadata differently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665904#comment-13665904
]
Hadoop QA commented on MAPREDUCE-5270:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584621/MAPREDUCE-5270.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3674//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3674//console
This message is automatically generated.
Migrate from using BuilderUtil factory methods to individual record factory
method on MapReduce side
Key: MAPREDUCE-5270
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch
Migrate the factory method on map reduce side.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated MAPREDUCE-5224:
---
Attachment: MAPREDUCE-5224.4.patch
Above comments have been addressed. Thanks.
BTW, I changed JobTracker#defaultFs back to fs, because some other codes in the
same package use this fs (fs was originally defined with no access modifier).
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.4.patch, MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
