[jira] [Commented] (MAPREDUCE-6070) yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
[
https://issues.apache.org/jira/browse/MAPREDUCE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131393#comment-14131393
]
Hudson commented on MAPREDUCE-6070:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #678 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/678/])
MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not
documented. Contributed by Tsuyoshi OZAWA (jlowe: rev
9e28e089c6bcb0b264751bdfa1f994ce002ef2de)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
--
Key: MAPREDUCE-6070
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6070
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.0, 2.5.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6070.2.patch, YARN-2078.1.patch
We should document the condition when uber mode is enabled. Currently, users
need to read following code to understand the condition.
{code}
boolean smallMemory =
( (Math.max(conf.getLong(MRJobConfig.MAP_MEMORY_MB, 0),
conf.getLong(MRJobConfig.REDUCE_MEMORY_MB, 0))
= sysMemSizeForUberSlot)
|| (sysMemSizeForUberSlot == JobConf.DISABLED_MEMORY_LIMIT));
boolean smallCpu =
Math.max(
conf.getInt(
MRJobConfig.MAP_CPU_VCORES,
MRJobConfig.DEFAULT_MAP_CPU_VCORES),
conf.getInt(
MRJobConfig.REDUCE_CPU_VCORES,
MRJobConfig.DEFAULT_REDUCE_CPU_VCORES))
= sysCPUSizeForUberSlot
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6070) yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
[
https://issues.apache.org/jira/browse/MAPREDUCE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131530#comment-14131530
]
Hudson commented on MAPREDUCE-6070:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1894 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1894/])
MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not
documented. Contributed by Tsuyoshi OZAWA (jlowe: rev
9e28e089c6bcb0b264751bdfa1f994ce002ef2de)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* hadoop-mapreduce-project/CHANGES.txt
yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
--
Key: MAPREDUCE-6070
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6070
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.0, 2.5.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6070.2.patch, YARN-2078.1.patch
We should document the condition when uber mode is enabled. Currently, users
need to read following code to understand the condition.
{code}
boolean smallMemory =
( (Math.max(conf.getLong(MRJobConfig.MAP_MEMORY_MB, 0),
conf.getLong(MRJobConfig.REDUCE_MEMORY_MB, 0))
= sysMemSizeForUberSlot)
|| (sysMemSizeForUberSlot == JobConf.DISABLED_MEMORY_LIMIT));
boolean smallCpu =
Math.max(
conf.getInt(
MRJobConfig.MAP_CPU_VCORES,
MRJobConfig.DEFAULT_MAP_CPU_VCORES),
conf.getInt(
MRJobConfig.REDUCE_CPU_VCORES,
MRJobConfig.DEFAULT_REDUCE_CPU_VCORES))
= sysCPUSizeForUberSlot
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6070) yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
[
https://issues.apache.org/jira/browse/MAPREDUCE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131562#comment-14131562
]
Hudson commented on MAPREDUCE-6070:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1869 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1869/])
MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not
documented. Contributed by Tsuyoshi OZAWA (jlowe: rev
9e28e089c6bcb0b264751bdfa1f994ce002ef2de)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
--
Key: MAPREDUCE-6070
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6070
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.0, 2.5.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6070.2.patch, YARN-2078.1.patch
We should document the condition when uber mode is enabled. Currently, users
need to read following code to understand the condition.
{code}
boolean smallMemory =
( (Math.max(conf.getLong(MRJobConfig.MAP_MEMORY_MB, 0),
conf.getLong(MRJobConfig.REDUCE_MEMORY_MB, 0))
= sysMemSizeForUberSlot)
|| (sysMemSizeForUberSlot == JobConf.DISABLED_MEMORY_LIMIT));
boolean smallCpu =
Math.max(
conf.getInt(
MRJobConfig.MAP_CPU_VCORES,
MRJobConfig.DEFAULT_MAP_CPU_VCORES),
conf.getInt(
MRJobConfig.REDUCE_CPU_VCORES,
MRJobConfig.DEFAULT_REDUCE_CPU_VCORES))
= sysCPUSizeForUberSlot
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to support any type path name.
zhihai xu created MAPREDUCE-6086: Summary: change mapreduce.job.credentials.binary configuration to support any type path name. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: zhihai xu Assignee: zhihai xu Change mapreduce.job.credentials.binary configuration to support any type path name. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to improve it to support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6086: - Description: Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. was: Change mapreduce.job.credentials.binary configuration to support any type path name. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to improve it to support non-local FS URIs. Issue Type: Bug (was: Improvement) Summary: change mapreduce.job.credentials.binary configuration to handle all URIs properly. (was: change mapreduce.job.credentials.binary configuration to support any type path name.) change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: zhihai xu Assignee: zhihai xu Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6086: - Issue Type: Improvement (was: Bug) change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: zhihai xu Assignee: zhihai xu Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6086: - Component/s: security change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security Reporter: zhihai xu Assignee: zhihai xu Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6086: - Affects Version/s: 2.5.0 change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6086: - Status: Patch Available (was: Open) change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6086.000.patch Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
Jian He created MAPREDUCE-6087: -- Summary: MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong Key: MAPREDUCE-6087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS now has double prefix as yarn.app.mapreduce. + yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6085) Facilitate processing of text files without key/value split
[ https://issues.apache.org/jira/browse/MAPREDUCE-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Sivachenko updated MAPREDUCE-6085: - Assignee: (was: Dmitry Sivachenko) Facilitate processing of text files without key/value split --- Key: MAPREDUCE-6085 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6085 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.4.1 Reporter: Dmitry Sivachenko Attachments: IdentifierResolver1.java.patch There is a rather popular type of task: processing of text files line by line without splitting line to key/value pair in streaming mode. (UNIX commands like grep, awk, etc, any filter scripts). By default, Hadoop streaming interface uses TextInputFormat which suites well for this task: it passes the input line itself to streaming job stdin. TextOutputReader class, which receives streaming job's output, splits it for key and value pair, and TextOutputFormat tries to merge this pair with separator. This results in extra separator appearing in the output in some cases. KeyOnlyTextOutputReader solves this problem: it passes the whole line as a key with null value, and TextOutputFormat correctly writes it without any separators inserted. I propose to add another IdentifierResolver: keyonlytextoutput, which uses standard TextInputWriter but replaces TextOutputReader with KeyOnlyTextOutputReader). As a result, lines of text are never split into key/value pair and never joined back, so lines appear in the output unmodified. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132384#comment-14132384
]
Hadoop QA commented on MAPREDUCE-6086:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668484/MAPREDUCE-6086.000.patch
against trunk revision a0ad975.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4871//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4871//console
This message is automatically generated.
change mapreduce.job.credentials.binary configuration to handle all URIs
properly.
Key: MAPREDUCE-6086
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security
Affects Versions: 2.5.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6086.000.patch
Change mapreduce.job.credentials.binary configuration to handle all URIs
properly.
The current mapreduce.job.credentials.binary configuration only support
local fs, It would be better to make it support non-local FS URIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[
https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132418#comment-14132418
]
Hadoop QA commented on MAPREDUCE-5657:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12621435/5657-trunk.patch
against trunk revision a0ad975.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4872//console
This message is automatically generated.
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
-
Key: MAPREDUCE-5657
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Attachments: 5657-branch-2.patch, 5657-branch-2.patch,
5657-trunk.patch, 5657-trunk.patch
Javadoc is more strict by default in JDK8 and will error out on malformed or
illegal tags found in doc comments. Although tagged as JDK8 all of the
required changes are generic Javadoc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6086) change mapreduce.job.credentials.binary configuration to handle all URIs properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132428#comment-14132428 ] zhihai xu commented on MAPREDUCE-6086: -- The test failure is because s3 Filesystem is not configured in TestMRCJCFileInputFormat. It is not related to my change. change mapreduce.job.credentials.binary configuration to handle all URIs properly. Key: MAPREDUCE-6086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6086.000.patch Change mapreduce.job.credentials.binary configuration to handle all URIs properly. The current mapreduce.job.credentials.binary configuration only support local fs, It would be better to make it support non-local FS URIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6088) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache.
zhihai xu created MAPREDUCE-6088: Summary: TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache. Key: MAPREDUCE-6088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6088 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, test Reporter: zhihai xu Assignee: zhihai xu TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it executed after testTokenCache. The reason is because testLocalJobTokenCache depends on the jConf setup by testTokenCache. The fix is to set up the JobConf separately for testLocalJobTokenCache and testTokenCache. See the following test result. Testcase: testLocalJobTokenCache took 1.197 sec FAILED local Job failed junit.framework.AssertionFailedError: local Job failed at org.apache.hadoop.mapreduce.security.TestTokenCache.testLocalJobTokenCache(TestTokenCache.java:258) Testcase: testGetTokensForNamenodes took 0.003 sec Testcase: testGetTokensForUriWithoutAuth took 0.01 sec Testcase: testCleanUpTokenReferral took 0.014 sec Testcase: testTokenCache took 13.316 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6088) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6088: - Status: Patch Available (was: Open) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache. - Key: MAPREDUCE-6088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6088 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, test Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6088.branch-1.0.patch TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it executed after testTokenCache. The reason is because testLocalJobTokenCache depends on the jConf setup by testTokenCache. The fix is to set up the JobConf separately for testLocalJobTokenCache and testTokenCache. See the following test result. Testcase: testLocalJobTokenCache took 1.197 sec FAILED local Job failed junit.framework.AssertionFailedError: local Job failed at org.apache.hadoop.mapreduce.security.TestTokenCache.testLocalJobTokenCache(TestTokenCache.java:258) Testcase: testGetTokensForNamenodes took 0.003 sec Testcase: testGetTokensForUriWithoutAuth took 0.01 sec Testcase: testCleanUpTokenReferral took 0.014 sec Testcase: testTokenCache took 13.316 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6088) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6088: - Attachment: MAPREDUCE-6088.branch-1.0.patch TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache. - Key: MAPREDUCE-6088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6088 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, test Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6088.branch-1.0.patch TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it executed after testTokenCache. The reason is because testLocalJobTokenCache depends on the jConf setup by testTokenCache. The fix is to set up the JobConf separately for testLocalJobTokenCache and testTokenCache. See the following test result. Testcase: testLocalJobTokenCache took 1.197 sec FAILED local Job failed junit.framework.AssertionFailedError: local Job failed at org.apache.hadoop.mapreduce.security.TestTokenCache.testLocalJobTokenCache(TestTokenCache.java:258) Testcase: testGetTokensForNamenodes took 0.003 sec Testcase: testGetTokensForUriWithoutAuth took 0.01 sec Testcase: testCleanUpTokenReferral took 0.014 sec Testcase: testTokenCache took 13.316 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6088) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132452#comment-14132452
]
Hadoop QA commented on MAPREDUCE-6088:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668534/MAPREDUCE-6088.branch-1.0.patch
against trunk revision 3c91817.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4873//console
This message is automatically generated.
TestTokenCache in mrv1 (branch-1) depend on the order of test execution.
testLocalJobTokenCache will fail if it execute after testTokenCache.
-
Key: MAPREDUCE-6088
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6088
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, test
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6088.branch-1.0.patch
TestTokenCache in mrv1 (branch-1) depend on the order of test execution.
testLocalJobTokenCache will fail if it executed after testTokenCache.
The reason is because testLocalJobTokenCache depends on the jConf setup by
testTokenCache.
The fix is to set up the JobConf separately for testLocalJobTokenCache and
testTokenCache.
See the following test result.
Testcase: testLocalJobTokenCache took 1.197 sec
FAILED
local Job failed
junit.framework.AssertionFailedError: local Job failed
at
org.apache.hadoop.mapreduce.security.TestTokenCache.testLocalJobTokenCache(TestTokenCache.java:258)
Testcase: testGetTokensForNamenodes took 0.003 sec
Testcase: testGetTokensForUriWithoutAuth took 0.01 sec
Testcase: testCleanUpTokenReferral took 0.014 sec
Testcase: testTokenCache took 13.316 sec
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2841) Task level native optimization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2841: --- Resolution: Fixed Fix Version/s: 3.0.0 Release Note: Adds a native implementation of the map output collector. The native library will build automatically with -Pnative. Users may choose the new collector on a job-by-job basis by setting mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred. nativetask.NativeMapOutputCollectorDelegator in their job configuration. For shuffle-intensive jobs this may provide speed-ups of 30% or more. was:Task level native optimization Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) The vote on the mailing list passed, so I merged the feature branch into trunk. Thanks to all of the contributors! Let's let this bake in trunk for a little while and consider a backport to branch-2 down the road if there is demand. Marking the issue as resolved for now. Task level native optimization -- Key: MAPREDUCE-2841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Environment: x86-64 Linux/Unix Reporter: Binglin Chang Assignee: Sean Zhong Fix For: 3.0.0 Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch, MAPREDUCE-2841.v2.patch, MR-2841benchmarks.pdf, dualpivot-0.patch, dualpivotv20-0.patch, fb-shuffle.patch, hadoop-3.0-mapreduce-2841-2014-7-17.patch, micro-benchmark.txt, mr-2841-merge-2.txt, mr-2841-merge-3.patch, mr-2841-merge-4.patch, mr-2841-merge.txt I'm recently working on native optimization for MapTask based on JNI. The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs emitted by mapper, therefore sort, spill, IFile serialization can all be done in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising results: 1. Sort is about 3x-10x as fast as java(only binary string compare is supported) 2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware CRC32C is used, things can get much faster(1G/ 3. Merge code is not completed yet, so the test use enough io.sort.mb to prevent mid-spill This leads to a total speed up of 2x~3x for the whole MapTask, if IdentityMapper(mapper does nothing) is used There are limitations of course, currently only Text and BytesWritable is supported, and I have not think through many things right now, such as how to support map side combine. I had some discussion with somebody familiar with hive, it seems that these limitations won't be much problem for Hive to benefit from those optimizations, at least. Advices or discussions about improving compatibility are most welcome:) Currently NativeMapOutputCollector has a static method called canEnable(), which checks if key/value type, comparator type, combiner are all compatible, then MapTask can choose to enable NativeMapOutputCollector. This is only a preliminary test, more work need to be done. I expect better final results, and I believe similar optimization can be adopt to reduce task and shuffle too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6088) TestTokenCache in mrv1 (branch-1) depend on the order of test execution. testLocalJobTokenCache will fail if it execute after testTokenCache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132461#comment-14132461
]
zhihai xu commented on MAPREDUCE-6088:
--
I uploaded a patch MAPREDUCE-6088.branch-1.0.patch for review.
With the patch, the test can pass without dependence:
Testcase: testCleanUpTokenReferral took 0.011 sec
Testcase: testLocalJobTokenCache took 14.09 sec
Testcase: testGetTokensForNamenodes took 0.004 sec
Testcase: testGetTokensForUriWithoutAuth took 0.007 sec
Testcase: testTokenCache took 12.949 sec
The root cause for the test ( testLocalJobTokenCache ) failure is the JobConf
used by testLocalJobTokenCache didn't configure mapred.job.tracker, so it
call LocalJobRunner which don't create job token.
see the following code in JobClient.java
{code}
String tracker = conf.get(mapred.job.tracker, local);
tasklogtimeout = conf.getInt(
TASKLOG_PULL_TIMEOUT_KEY, DEFAULT_TASKLOG_TIMEOUT);
this.ugi = UserGroupInformation.getCurrentUser();
if (local.equals(tracker)) {
conf.setNumMapTasks(1);
this.jobSubmitClient = new LocalJobRunner(conf);
} else {
this.rpcJobSubmitClient =
createRPCProxy(JobTracker.getAddress(conf), conf);
this.jobSubmitClient = createProxy(this.rpcJobSubmitClient, conf);
}
{code}
TestTokenCache in mrv1 (branch-1) depend on the order of test execution.
testLocalJobTokenCache will fail if it execute after testTokenCache.
-
Key: MAPREDUCE-6088
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6088
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, test
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6088.branch-1.0.patch
TestTokenCache in mrv1 (branch-1) depend on the order of test execution.
testLocalJobTokenCache will fail if it executed after testTokenCache.
The reason is because testLocalJobTokenCache depends on the jConf setup by
testTokenCache.
The fix is to set up the JobConf separately for testLocalJobTokenCache and
testTokenCache.
See the following test result.
Testcase: testLocalJobTokenCache took 1.197 sec
FAILED
local Job failed
junit.framework.AssertionFailedError: local Job failed
at
org.apache.hadoop.mapreduce.security.TestTokenCache.testLocalJobTokenCache(TestTokenCache.java:258)
Testcase: testGetTokensForNamenodes took 0.003 sec
Testcase: testGetTokensForUriWithoutAuth took 0.01 sec
Testcase: testCleanUpTokenReferral took 0.014 sec
Testcase: testTokenCache took 13.316 sec
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
