[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555874#comment-14555874 ] Tsuyoshi Ozawa commented on MAPREDUCE-6204: --- +1, committing this shortly. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548155#comment-14548155
]
Tsuyoshi Ozawa commented on MAPREDUCE-6204:
---
[~sam liu], thank you for pinging us. Let me clarify one point: the
configurations as follows looks set by your configurations as [~jira.shegalov]
mentioned. Is it right?
{code}
-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized
-Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal
-Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt
-Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch,
MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549752#comment-14549752 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, In fact, there is no hadoop cluster and hadoop related configuration in my dev env on power linux, so I also did not sure why the 'MAPRED_MAP_TASK_JAVA_OPTS' and 'MAPRED_REDUCE_TASK_JAVA_OPTS' has above default and unexpected value there. At the same time, we mentioned that the root causes of the issue is described in MAPREDUCE-6205, however its patch is still not reviewed/accepted yet. Therefore, current fix will greatly make sense to the ut TestJobCounters: it explicitly replaces the deprecated property with the legal ones, and really could set correct value to MAP/REDUCE OPTS properties and fix the unexpected env/configurations issue and make the ut more robust. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549762#comment-14549762 ] Tsuyoshi Ozawa commented on MAPREDUCE-6204: --- OK. I agree with the fix for using newer properties instead of using deprecated one. [~jira.shegalov] As you mentioned, the test failure can be not related and it's addressed on MAPREDUCE-6205. However, we should move using newer, we can say more proper, properties. Do you agree with fixing this? TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543278#comment-14543278 ] sam liu commented on MAPREDUCE-6204: It's really weird that there always be some ut failures, however none of them was brought by current fix and all the ut could pass in my env. I guess there might be some env issues. I suggest to rerun the ut or deliver this fix directly after reviewing. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539675#comment-14539675
]
Hadoop QA commented on MAPREDUCE-6204:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 11s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | mapreduce tests | 99m 0s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 114m 11s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732201/MAPREDUCE-6204-3.patch
|
| Optional Tests | javac unit |
| git revision | trunk / 360dff5 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5712/console |
This message was automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch,
MAPREDUCE-6204-3.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541398#comment-14541398
]
Hadoop QA commented on MAPREDUCE-6204:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 22s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | mapreduce tests | 89m 27s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 104m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapred.TestLineRecordReaderJobs |
| | hadoop.mapreduce.security.TestBinaryTokenFile |
| | hadoop.mapred.TestCombineSequenceFileInputFormat |
| | hadoop.mapreduce.TestMapReduceLazyOutput |
| | hadoop.mapreduce.TestMRJobClient |
| Timed out tests | org.apache.hadoop.mapreduce.TestLargeSort |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732465/MAPREDUCE-6204-4.patch
|
| Optional Tests | javac unit |
| git revision | trunk / 2463666 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5721/console |
This message was automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch,
MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537761#comment-14537761 ] sam liu commented on MAPREDUCE-6204: These ut should not fail, as they could pass in my env. I studied the outputs of ut execution and seems the failures were caused by env issue, like 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException'. Could please help rerun the ut? Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530452#comment-14530452
]
Hadoop QA commented on MAPREDUCE-6204:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 15s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | mapreduce tests | 94m 49s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 110m 12s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestJobCounters |
| | hadoop.mapred.TestJobSysDirWithDFS |
| | hadoop.mapred.TestReduceFetchFromPartialMem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730785/MAPREDUCE-6204-2.patch
|
| Optional Tests | javac unit |
| git revision | trunk / a583a40 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5650/console |
This message was automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch,
MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525706#comment-14525706
]
Hadoop QA commented on MAPREDUCE-6204:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 5s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 40s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 104m 31s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 120m 51s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / a319771 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5624/console |
This message was automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525952#comment-14525952 ] Hadoop QA commented on MAPREDUCE-6204: -- (!) The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5629/ may provide some hints. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260057#comment-14260057
]
Gera Shegalov commented on MAPREDUCE-6204:
--
You are correct that mapred.child.java.opts should be added to the deprecation
deltas. Let us continue working on this in MAPREDUCE-6205.
However, this is not the root cause for the failure you are seeing on your
ppc64. You are just working around the issue that you somewhere have this
config that definitely does not belong to a clean build:
{code}
-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized
-Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal
-Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt
-Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260061#comment-14260061 ] Gera Shegalov commented on MAPREDUCE-6204: -- bq. I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() ... These properties are actually meant to be used by the MRAppMaster. So although they are accessible in task attempt JVM's, it's not really correct to verify them there when they play no role. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260694#comment-14260694 ] sam liu commented on MAPREDUCE-6204: Gera, thanks for your comments and I will update the patch of MAPREDUCE-6205 according to your suggestion later. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259797#comment-14259797 ] sam liu commented on MAPREDUCE-6204: For many deprecated properties, hadoop code always set its value to its new version of property. Absolutely, this solution is clear and won't bring any misunderstanding on the conception of 'deprecation'. However, for the deprecated property mapred.child.java.opts, hadoop uses a different method: it won't update the value of its new versions MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Yes, finally hadoop will set a correct value on the classpath, however the program like MapReduce could not get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS: it definately bring confusing to the users. Furthermore, this ut will fail without this patch on ppc64 platform, but could pass on the same env with this patch. Actually the root cause is the insuffcient completion of the deprecation of property mapred.child.java.opts. We could not say this patch is useless or simply pass the buck to 'env issue': if this is an env issue, this ut should fail on the ppc64 platform either with the deprecated property mapred.child.java.opts or with the new version of properties MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Therefore this patch is necessary and could remedy the insuffcient completion of the deprecation of property mapred.child.java.opts. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258689#comment-14258689 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- I'll commit latest patch in this weekend if there is no opposition. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258745#comment-14258745 ] Gera Shegalov commented on MAPREDUCE-6204: -- +0. The patch is not wrong but it has zero effect on a clean environment. In contrast, clearly, in [~sam liu]'s ppc64 environment there is a config file on the classpath defining non-default map.java.opts. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258164#comment-14258164 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- Sam, thanks for your clarification. +1 against MAPREDUCE-6204-1.patch with following reasons: 1. We should use new properties instead of deprecated one. 2. On some environment, JVM options are configured wrongly. As a result, TestJobCounters can fails. The patch fix the problem. [~jira.shegalov] please let me know if you have opinions. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256804#comment-14256804
]
Hadoop QA commented on MAPREDUCE-6204:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688804/MAPREDUCE-6204-1.patch
against trunk revision 5caebba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5089//console
This message is automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256900#comment-14256900
]
Tsuyoshi OZAWA commented on MAPREDUCE-6204:
---
I checked MapReduceChildJVM#getChildJavaOpts as Gera mentioned. The method
includes following code:
{code}
if (isMapTask) {
userClasspath =
jobConf.get(
JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
} else {
userClasspath =
jobConf.get(
JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
}
{code}
It means that MAPRED_TASK_JAVA_OPTS is used instead of
MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS if
MAPRED_TASK_JAVA_OPTS is set.
{quote}
MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G
MAPRED_MAP_TASK_JAVA_OPTS=null
MAPRED_REDUCE_TASK_JAVA_OPTS=null
{quote}
This looks correct behavior since MapReduceChildJVM#getChildJavaOpts just gets
the value of MAPRED_TASK_JAVA_OPTS instead of MAPRED_MAP_TASK_JAVA_OPTS and
MAPRED_REDUCE_TASK_JAVA_OPTS. I think the patch doesn't fix the problem. What
do you think? How about making MAPRED_TASK_JAVA_OPTS larger heap size?
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257775#comment-14257775 ] sam liu commented on MAPREDUCE-6204: I did many tests and it does not work if only making MAPRED_TASK_JAVA_OPTS larger heap size. Actually, MapReduceChildJVM#getChildJavaOpts is only be invoked by method MapReduceChildJVM#getVMCommand which is not be invoked by any other existing hadoop code: that's why in my tests the value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS is still null and not be the same as MAPRED_TASK_JAVA_OPTS. Only I set correct value to MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, the map/reduce tasks could get correct java opts as below and this unit test could pass on PPC64 platform: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257800#comment-14257800
]
Gera Shegalov commented on MAPREDUCE-6204:
--
getVMCommand is invoked to build the launch context in
{{o.a.h.mapreduce.v2.app.job.impl.TaskAttemptImpl#createContainerLaunchContext}}.
If you modify the test to set the DEBUG log level for MRAppMaster you will
even find the launch context printed in its syslog.
In TestJobCounters#runHeapUsageTestJob add
{code}
jobConf.set(MRJobConfig.MR_AM_LOG_LEVEL, DEBUG);
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257845#comment-14257845 ] sam liu commented on MAPREDUCE-6204: Gera, I see, but could you please help explain why the map/reduce tasks could not get expected value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, but only return 'null' in the test? I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() and TestJobCounters#MemoryLoaderReducer#configure(): System.out.println(MAPRED_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_TASK_JAVA_OPTS)); System.out.println(MAPRED_MAP_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS)); System.out.println(MAPRED_REDUCE_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS)); TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257866#comment-14257866
]
Tsuyoshi OZAWA commented on MAPREDUCE-6204:
---
Sam,
Note that
JobConf.MAPRED_MAP_TASK_JAVA_OPTS/JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS are not
updated when MapReduceChildJVM#getChildJavaOpts is called. Please check the
method:
{code}
if (isMapTask) {
userClasspath =
jobConf.get(
JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
} else {
userClasspath =
jobConf.get(
JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
...
}
{code}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257920#comment-14257920 ] sam liu commented on MAPREDUCE-6204: Tsuyoshi, Yes, you are correct and the two properties won't be updated. I did a test: 1. In TestTaskAttemptContainerRequest.java, I added 'jobConf.set(JobConf.MAPRED_TASK_JAVA_OPTS, -Xms32m -Xmx1G)' 2. Then I found the command args created by 'ContainerLaunchContext launchCtx = TaskAttemptImpl.createContainerLaunchContext...' likes: =$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xms32m -Xmx1G -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=LOG_DIR -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 0 attempt_1_0001_m_01_1 0 1LOG_DIR/stdout 2LOG_DIR/stderr Please mention that only a commn heap setting -Xms32m -Xmx1G is added into the command args, but the two properties(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS) were not updated as expected. So I think it's still necessary to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS values in TestJobCounters, as that test could pass on PPC64 only with such updates. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258009#comment-14258009 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- Basically, +1 for the fix since we should use new properties instead of deprecated one. My question, maybe Gera also has, is why the test could pass on PPC64 with same JVM options? I'd like to clarify it since it can be another bug if you can see JVM crash without your patch. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258086#comment-14258086 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I did more tests on PPC64, it has some difference from x86: 1. Without any change, the map/reduce tasks will print: 1.1 on x86: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS==null MAPRED_REDUCE_TASK_JAVA_OPTS=null 1.2 on PPC64: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd MAPRED_REDUCE_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd 2. With this patch, both x86 and PPC64 will print same message: MAPRED_TASK_JAVA_OPTS=-Xmx200m MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G I will look into PPC64 part to see if there is any other issue there. I also opened another jira MAPREDUCE-6205 to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS with MAPRED_TASK_JAVA_OPTS together, but the patch brought another new ut failure. I also will look into that jira later. So this ut really need this patch on PPC64 now and as you said we should use new properties instead of deprecated one. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256575#comment-14256575 ] sam liu commented on MAPREDUCE-6204: In Hadoop 2.x, the property JobConf.MAPRED_TASK_JAVA_OPTS is deprecated, and hadoop will ignore this property directly without replacing its new values(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS). This issue will cause test TestJobCounters fail on PPC64 platform sometimes, as PPC64 JVM sometimes comsumes more memory when executing map/reduce tasks and might encounter JVM crashing issue once the default heap size is exhausted. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256579#comment-14256579
]
Hadoop QA commented on MAPREDUCE-6204:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688787/MAPREDUCE-6204.patch
against trunk revision fdf042d.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5088//console
This message is automatically generated.
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256594#comment-14256594 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- [~sam liu], the fix itself good to me, but the patch is not valid one. Could you generate the patch by using git diff --no-prefix command? TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256604#comment-14256604
]
Gera Shegalov commented on MAPREDUCE-6204:
--
I am not sure it helps with crashes because memory settings have not changed
with this patch if you look at
{{MapReduceChildJVM#getChildJavaOpts}}
TestJobCounters should use new properties instead
JobConf.MAPRED_TASK_JAVA_OPTS
---
Key: MAPREDUCE-6204
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6204.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256687#comment-14256687 ] sam liu commented on MAPREDUCE-6204: Gera, I did a test: 1. Without the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get value 'null' for properties MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS during MR job execution: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=null MAPRED_REDUCE_TASK_JAVA_OPTS=null 2. With the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get correct value for all the properties: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256718#comment-14256718 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I attached new patch MAPREDUCE-6204-1.patch and could you please help take a review? Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
