[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256804#comment-14256804
]
Hadoop QA commented on MAPREDUCE-6204:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch
against trunk revision 5caebba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//console
This message is automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256900#comment-14256900
]
Tsuyoshi OZAWA commented on MAPREDUCE-6204:
---
I checked MapReduceChildJVM#getChildJavaOpts as Gera mentioned. The method
includes following code:
{code}
if (isMapTask) {
userClasspath =
jobConf.get(
JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
} else {
userClasspath =
jobConf.get(
JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
}
{code}
It means that MAPRED_TASK_JAVA_OPTS is used instead of
MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS if
MAPRED_TASK_JAVA_OPTS is set.
{quote}
MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
MAPRED_MAP_TASK_JAVA_OPTS=null
MAPRED_REDUCE_TASK_JAVA_OPTS=null
{quote}
This looks correct behavior since MapReduceChildJVM#getChildJavaOpts just gets
the value of MAPRED_TASK_JAVA_OPTS instead of MAPRED_MAP_TASK_JAVA_OPTS and
MAPRED_REDUCE_TASK_JAVA_OPTS. I think the patch doesn't fix the problem. What
do you think? How about making MAPRED_TASK_JAVA_OPTS larger heap size?
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[
https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257608#comment-14257608
]
Gera Shegalov commented on MAPREDUCE-4815:
--
I think we should strive for a solution that does not create any sibling
directories as it will surprise users, and it would mean that checkOutputSpec
everywhere needs to be adjusted in derived classes. I think we can modify the
behavior of the FOC based on [~l201514]'s idea but still use the existing
directory structure for backwards-compatibility:
task attempts write as usual to
$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
{{commitTask}}:
# rename *all* files
'$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/foo' to
'$joboutput/foo'.
# rename '$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID' to
'$joboutput/_temporary/$appAttemptID/$taskID' , which is the actual commit
{{recoverTask}}:
# if '$joboutput/_temporary/$(appAttemptID - 1)/$taskID' exists: rename to
'$joboutput/_temporary/$appAttemptID/$taskID'
# for backwards compatibility after upgrade to the new logic, check if there
are any '$joboutput/_temporary/$appAttemptID/$taskID/foo' and rename them to
'$joboutput/foo'
{{commitJob}}
# blow away $joboutput/_temporary
# write $joboutput/_SUCCESS
FileOutputCommitter.commitJob can be very slow for jobs with many output files
--
Key: MAPREDUCE-4815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1
Reporter: Jason Lowe
Assignee: Siqi Li
Attachments: MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch,
MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch,
MAPREDUCE-4815.v8.patch
If a job generates many files to commit then the commitJob method call at the
end of the job can take minutes. This is a performance regression from 1.x,
as 1.x had the tasks commit directly to the final output directory as they
were completing and commitJob had very little to do. The commit work was
processed in parallel and overlapped the processing of outstanding tasks. In
0.23/2.x, the commit is single-threaded and waits until all tasks have
completed before commencing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257775#comment-14257775 ] sam liu commented on MAPREDUCE-6204: I did many tests and it does not work if only making MAPRED_TASK_JAVA_OPTS larger heap size. Actually, MapReduceChildJVM#getChildJavaOpts is only be invoked by method MapReduceChildJVM#getVMCommand which is not be invoked by any other existing hadoop code: that's why in my tests the value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS is still null and not be the same as MAPRED_TASK_JAVA_OPTS. Only I set correct value to MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, the map/reduce tasks could get correct java opts as below and this unit test could pass on PPC64 platform: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257800#comment-14257800
]
Gera Shegalov commented on MAPREDUCE-6204:
--
getVMCommand is invoked to build the launch context in
{{o.a.h.mapreduce.v2.app.job.impl.TaskAttemptImpl#createContainerLaunchContext}}.
If you modify the test to set the DEBUG log level for MRAppMaster you will
even find the launch context printed in its syslog.
In TestJobCounters#runHeapUsageTestJob add
{code}
jobConf.set(MRJobConfig.MR_AM_LOG_LEVEL, DEBUG);
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257845#comment-14257845 ] sam liu commented on MAPREDUCE-6204: Gera, I see, but could you please help explain why the map/reduce tasks could not get expected value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, but only return 'null' in the test? I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() and TestJobCounters#MemoryLoaderReducer#configure(): System.out.println(MAPRED_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_TASK_JAVA_OPTS)); System.out.println(MAPRED_MAP_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS)); System.out.println(MAPRED_REDUCE_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS)); TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257866#comment-14257866
]
Tsuyoshi OZAWA commented on MAPREDUCE-6204:
---
Sam,
Note that
JobConf.MAPRED_MAP_TASK_JAVA_OPTS/JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS are not
updated when MapReduceChildJVM#getChildJavaOpts is called. Please check the
method:
{code}
if (isMapTask) {
userClasspath =
jobConf.get(
JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
} else {
userClasspath =
jobConf.get(
JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
}
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257920#comment-14257920 ] sam liu commented on MAPREDUCE-6204: Tsuyoshi, Yes, you are correct and the two properties won't be updated. I did a test: 1. In TestTaskAttemptContainerRequest.java, I added 'jobConf.set(JobConf.MAPRED_TASK_JAVA_OPTS, -Xms32m -Xmx1G)' 2. Then I found the command args created by 'ContainerLaunchContext launchCtx = TaskAttemptImpl.createContainerLaunchContext...' likes: =$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xms32m -Xmx1G -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=LOG_DIR -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 0 attempt_1_0001_m_01_1 0 1LOG_DIR/stdout 2LOG_DIR/stderr Please mention that only a commn heap setting -Xms32m -Xmx1G is added into the command args, but the two properties(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS) were not updated as expected. So I think it's still necessary to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS values in TestJobCounters, as that test could pass on PPC64 only with such updates. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
sam liu created MAPREDUCE-6205: -- Summary: Updated the value of the new version properties of the deprecated property mapred.child.java.opts Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: sam liu Assignee: sam liu Priority: Minor In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Attachment: MAPREDUCE-6205.patch Unlike many other deprecated properties, the old property mapred.child.java.opts does not only has one new version property, but has two new version properties: MRJobConfig.MAP_JAVA_OPTS, MRJobConfig.REDUCE_JAVA_OPTS. So we should add a new function Configuration#DeprecationDelta(String key, String[] newKeys) to support it. Updated the value of the new version properties of the deprecated property mapred.child.java.opts --- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Affects Version/s: trunk Status: Patch Available (was: Open) If we only set value -Xms32m -Xmx1G to the deprecated property mapred.child.java.opts before MR job execution: - Without this patch, during map/reduce tasks execution, we can NOT get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, but get null value: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.MAP_JAVA_OPTS=null MRJobConfig.REDUCE_JAVA_OPTS=null - With this patch, during map/reduce tasks execution, we can get correct value of them: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.MAP_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.REDUCE_JAVA_OPTS=-Xms32m -Xmx1G Updated the value of the new version properties of the deprecated property mapred.child.java.opts --- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257997#comment-14257997
]
Hadoop QA commented on MAPREDUCE-6205:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688990/MAPREDUCE-6205.patch
against trunk revision 41a548a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 16 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapred.TestJobConf
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5090//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5090//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5090//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5090//console
This message is automatically generated.
Updated the value of the new version properties of the deprecated property
mapred.child.java.opts
---
Key: MAPREDUCE-6205
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6205.patch
In current hadoop code, the old property mapred.child.java.opts is
deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
deprecated property mapred.child.java.opts, hadoop won't automatically
update its new versions properties
MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and
MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will
update the new version properties for many other deprecated properties, we
also should support such feature on the old property
mapred.child.java.opts, otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Summary: Update the value of the new version properties of the deprecated property mapred.child.java.opts (was: Updated the value of the new version properties of the deprecated property mapred.child.java.opts) Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258009#comment-14258009 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- Basically, +1 for the fix since we should use new properties instead of deprecated one. My question, maybe Gera also has, is why the test could pass on PPC64 with same JVM options? I'd like to clarify it since it can be another bug if you can see JVM crash without your patch. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258086#comment-14258086 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I did more tests on PPC64, it has some difference from x86: 1. Without any change, the map/reduce tasks will print: 1.1 on x86: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS==null MAPRED_REDUCE_TASK_JAVA_OPTS=null 1.2 on PPC64: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd MAPRED_REDUCE_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd 2. With this patch, both x86 and PPC64 will print same message: MAPRED_TASK_JAVA_OPTS=-Xmx200m MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G I will look into PPC64 part to see if there is any other issue there. I also opened another jira MAPREDUCE-6205 to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS with MAPRED_TASK_JAVA_OPTS together, but the patch brought another new ut failure. I also will look into that jira later. So this ut really need this patch on PPC64 now and as you said we should use new properties instead of deprecated one. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
