[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Status: Open (was: Patch Available)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Attachment: MAPREDUCE-5957.patch
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Status: Patch Available (was: Open)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063296#comment-14063296
]
Hadoop QA commented on MAPREDUCE-5957:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656001/MAPREDUCE-5957.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4740//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4740//console
This message is automatically generated.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3418: --- Assignee: Chen He If map output is not found, shuffle runs in tight loop -- Key: MAPREDUCE-3418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 2.3.0 Reporter: John George Assignee: Chen He Sharad Agarwal bumped into this while simulating fetch failures. Removed the map output directory. Shuffle runs in tight loop throwing : 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063557#comment-14063557 ] Rohith commented on MAPREDUCE-5910: --- Thanks [~jianhe] for reviewing the patch. I will update patch ASAP :-) MRAppMaster should handle Resync from RM instead of shutting down. -- Key: MAPREDUCE-5910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The MRAppMaster behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-5910: -- Attachment: MAPREDUCE-5910.3.patch I updated patch for above 2 [~jianhe]'s comments.Pleae review updated patch. MRAppMaster should handle Resync from RM instead of shutting down. -- Key: MAPREDUCE-5910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch, MAPREDUCE-5910.3.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The MRAppMaster behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063625#comment-14063625
]
Sangjin Lee commented on MAPREDUCE-5957:
The thread context classloader is what makes this type of problems difficult to
fix completely. The thread context classloader is problematic in so many ways.
It has no formal contract defined, and it represents an ultimate backdoor for
thwarting any modularity.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
Charles Lamb created MAPREDUCE-5971: --- Summary: move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated MAPREDUCE-5971: Status: Patch Available (was: Open) A simple patch is attached. I didn't make any unit tests since TestDistCpWithXAttrs exercises this. I can easily add a specific unit test for this though. move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063747#comment-14063747 ] Jian He commented on MAPREDUCE-5910: Thanks for updating the patch! patch looks good. submit to jenkins MRAppMaster should handle Resync from RM instead of shutting down. -- Key: MAPREDUCE-5910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch, MAPREDUCE-5910.3.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The MRAppMaster behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5910: --- Status: Patch Available (was: Open) MRAppMaster should handle Resync from RM instead of shutting down. -- Key: MAPREDUCE-5910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch, MAPREDUCE-5910.3.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The MRAppMaster behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063797#comment-14063797
]
Hadoop QA commented on MAPREDUCE-5910:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656052/MAPREDUCE-5910.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4741//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4741//console
This message is automatically generated.
MRAppMaster should handle Resync from RM instead of shutting down.
--
Key: MAPREDUCE-5910
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910
Project: Hadoop Map/Reduce
Issue Type: Task
Components: applicationmaster
Reporter: Rohith
Assignee: Rohith
Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch,
MAPREDUCE-5910.3.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The MRAppMaster behavior is expected to change
to calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated MAPREDUCE-5971: Attachment: MAPREDUCE-5971.001.patch move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Attachments: MAPREDUCE-5971.001.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063917#comment-14063917 ] Andrew Wang commented on MAPREDUCE-5971: Hi Charles, thanks for working on this, I took a quick look: Since we're adding a new getDefaultValue to DistCpOptionSwitch, shouldn't we make the handling of default values in CustomParser generic as well? Right now using the default value is still special cased only for PRESERVE_STATUS. Maybe build a map with the default values in CustomParser? move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Attachments: MAPREDUCE-5971.001.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5972) Fix typo 'programatically' in job.xml
Akira AJISAKA created MAPREDUCE-5972:
Summary: Fix typo 'programatically' in job.xml
Key: MAPREDUCE-5972
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5972
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
In job.xml, there's a typo 'programatically' as the below if a property is set
through program.
{code}
property
namemapreduce.job.map.class/name
valueorg.apache.hadoop.examples.WordCount$TokenizerMapper/value
sourceprogramatically/source
/property
{code}
should be 'programmatically'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5972) Fix typo 'programatically' in job.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5972:
-
Target Version/s: 2.6.0
Status: Patch Available (was: Open)
Fix typo 'programatically' in job.xml
-
Key: MAPREDUCE-5972
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5972
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5972.patch
In job.xml, there's a typo 'programatically' as the below if a property is
set through program.
{code}
property
namemapreduce.job.map.class/name
valueorg.apache.hadoop.examples.WordCount$TokenizerMapper/value
sourceprogramatically/source
/property
{code}
should be 'programmatically'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5972) Fix typo 'programatically' in job.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5972:
-
Attachment: MAPREDUCE-5972.patch
Fix typo 'programatically' in job.xml
-
Key: MAPREDUCE-5972
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5972
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5972.patch
In job.xml, there's a typo 'programatically' as the below if a property is
set through program.
{code}
property
namemapreduce.job.map.class/name
valueorg.apache.hadoop.examples.WordCount$TokenizerMapper/value
sourceprogramatically/source
/property
{code}
should be 'programmatically'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5910) MRAppMaster should handle Resync from RM instead of shutting down.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063985#comment-14063985 ] Jian He commented on MAPREDUCE-5910: Rohith, can you look into the test failures? thanks! MRAppMaster should handle Resync from RM instead of shutting down. -- Key: MAPREDUCE-5910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5910 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5910.1.patch, MAPREDUCE-5910.2.patch, MAPREDUCE-5910.3.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The MRAppMaster behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063995#comment-14063995
]
Hadoop QA commented on MAPREDUCE-5971:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656099/MAPREDUCE-5971.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4742//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4742//console
This message is automatically generated.
move the default options for distcp -p to DistCpOptionSwitch
Key: MAPREDUCE-5971
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distcp
Affects Versions: trunk
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
Attachments: MAPREDUCE-5971.001.patch
The default preserve flags for distcp -p are embedded in the OptionsParser
code. Refactor to co-locate them with the actual flag initialization.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064044#comment-14064044
]
Jason Lowe commented on MAPREDUCE-5957:
---
Thanks for updating the patch. I do think the wrapper makes it a bit clearer
we're calling something within the context of the job classloader.
One of the drawbacks of the wrapper is that it's harder to deal with checked
exceptions. For example, MRAppMaster.isRecoverySupported used to propagate any
IOException thrown by the user-provided committer code, but the wrapper now
transforms this into a runtime exception. Not sure it really makes a whole lot
of difference in practice for that case, but it is a subtle change in behavior.
Do you think it's worth adding a wrapper form that can throw Exception (or
maybe just IOException)? Thinking along similar lines as PrivilegedAction vs.
PrivilegedExceptionAction in the doAs scenarios. I guess callers can catch
YarnRuntimeException with the current patch and check the cause if they
want/need to convert it back to a checked exception themselves, but curious on
your thoughts.
Otherwise I think the patch looks good. Couple of minor additional items:
- Is there a reason the init transition calls MRAppMaster.createJobClassloader
rather than having MRAppMaster.serviceInit do it? Seems more natural to have
the serviceInit method initialize the MRAppMaster data members if possible.
- Not strictly related to this JIRA, but I happened to notice that
MRAppMaster.isRecoverySupported takes a committer argument that it totally
ignores. Since we're in the area, it would be nice to clean that up by either
removing the argument or use it rather than the committer member. Totally OK
with me if you'd rather postpone that minor cleanup to another JIRA.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5952) LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex
[
https://issues.apache.org/jira/browse/MAPREDUCE-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064114#comment-14064114
]
Karthik Kambatla commented on MAPREDUCE-5952:
-
Thanks [~jira.shegalov]. Missed the fact that the method was moved out of the
class. I agree with you making it static is a better approach; i.e.,
MAPREDUCE-5952.v03.patch.
+1. Committing this shortly.
LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single
dir for mapOutIndex
Key: MAPREDUCE-5952
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5952
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5952.v01.patch, MAPREDUCE-5952.v02.patch,
MAPREDUCE-5952.v03.patch, MAPREDUCE-5952.v04.patch
The javadoc comment for {{renameMapOutputForReduce}} incorrectly refers to a
single map output directory, whereas this depends on LOCAL_DIRS.
mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()
{code}
2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
(uberized) 'child' : java.io.FileNotFoundException: File
/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.
TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)
at
org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5952) LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex
[
https://issues.apache.org/jira/browse/MAPREDUCE-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5952:
Resolution: Fixed
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Gera. Committed this to trunk, branch-2 and branch-2.5.
LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single
dir for mapOutIndex
Key: MAPREDUCE-5952
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5952
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Fix For: 2.5.0
Attachments: MAPREDUCE-5952.v01.patch, MAPREDUCE-5952.v02.patch,
MAPREDUCE-5952.v03.patch, MAPREDUCE-5952.v04.patch
The javadoc comment for {{renameMapOutputForReduce}} incorrectly refers to a
single map output directory, whereas this depends on LOCAL_DIRS.
mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()
{code}
2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
(uberized) 'child' : java.io.FileNotFoundException: File
/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.
TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)
at
org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5952) LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex
[
https://issues.apache.org/jira/browse/MAPREDUCE-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064141#comment-14064141
]
Hudson commented on MAPREDUCE-5952:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #5894 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5894/])
MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly
assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611196)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single
dir for mapOutIndex
Key: MAPREDUCE-5952
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5952
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Fix For: 2.5.0
Attachments: MAPREDUCE-5952.v01.patch, MAPREDUCE-5952.v02.patch,
MAPREDUCE-5952.v03.patch, MAPREDUCE-5952.v04.patch
The javadoc comment for {{renameMapOutputForReduce}} incorrectly refers to a
single map output directory, whereas this depends on LOCAL_DIRS.
mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()
{code}
2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
(uberized) 'child' : java.io.FileNotFoundException: File
/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.
TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)
at
org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064149#comment-14064149
]
Sangjin Lee commented on MAPREDUCE-5957:
bq. Do you think it's worth adding a wrapper form that can throw Exception (or
maybe just IOException)?
Sure. I'll add an overloaded method that can throw an IOException. I did notice
that when I was playing with the wrapper method, and was thinking about adding
another version.
bq. Is there a reason the init transition calls
MRAppMaster.createJobClassloader rather than having MRAppMaster.serviceInit do
it?
Nothing other than that I wanted to create it as early as possible. There is a
fair amount of code between the beginning of that method and serviceInit(). But
I think it is fine to do it inside MRAppMaster.serviceInit(). I'll change that.
bq. it would be nice to clean that up by either removing the argument or use it
rather than the committer member.
I'd be happy to do it. I'll fix that too.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated MAPREDUCE-5971: Attachment: MAPREDUCE-5971.002.patch Thanks for the review [~andrew.wang]. Per our conversation offline, here's the dead-simple change (move it to a constant). move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064171#comment-14064171 ] Andrew Wang commented on MAPREDUCE-5971: +1 pending, thanks charles move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5672) Provide optional RollingFileAppender for container log4j (syslog)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-5672: - Assignee: Gera Shegalov (was: Andy Isaacson) Provide optional RollingFileAppender for container log4j (syslog) - Key: MAPREDUCE-5672 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 2.2.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Fix For: 2.3.0 Attachments: MAPREDUCE-5672.v01.patch, MAPREDUCE-5672.v02.patch, MAPREDUCE-5672.v03.patch, MAPREDUCE-5672.v04.patch, MAPREDUCE-5672.v05.patch, MAPREDUCE-5672.v06.patch, MAPREDUCE-5672.v07.patch, MAPREDUCE-5672.v07.patch, Screen Shot 2013-12-05 at 3.21.02 PM.png, Screen Shot 2013-12-05 at 3.23.33 PM.png This JIRA is an alternative take on YARN-1130 We propose providing an option of using a RollingFileAppender(RFA)-based implementation of container log appender as means of log size control via mapreduce.task.userlog.limit.kb. The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA. In addition yarn.app.mapreduce.container.log.backups (task attempt containers) and yarn.app.mapreduce.am.log.backups (MR-AM) are passed as maxBackupIndex. Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender (CRLA) co-exist. CLA is the default. CRLA is chosen when mapreduce.task.userlog.limit.kb 0 *.backups 0. Pros: 1) CRLA output is visible in UI right away. CLA output with mapreduce.task.userlog.limit.kb 0 is not visible until the task attempt finishes that prevents timely diagnostics. 2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb, no space is taken from the JVM heap. 3) No UI impact, since YARN is already designed to deal with any log name beyond stderr/out, syslog, debug.out, profile.out Cons: 1) if the logging is excessive there will be more local filesystem metadata I/O due to roll. That should be negligible in the grand scheme. Furthermore, to improve log consistency and completeness in the case of JVM crashes and SIGTERMing by NM, we propose to restore the MRv1 behavior of periodic log syncing (every 5s) and having log sync as part of a shutdown hook. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5971) move the default options for distcp -p to DistCpOptionSwitch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064238#comment-14064238
]
Hadoop QA commented on MAPREDUCE-5971:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656146/MAPREDUCE-5971.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4744//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4744//console
This message is automatically generated.
move the default options for distcp -p to DistCpOptionSwitch
Key: MAPREDUCE-5971
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distcp
Affects Versions: trunk
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch
The default preserve flags for distcp -p are embedded in the OptionsParser
code. Refactor to co-locate them with the actual flag initialization.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5972) Fix typo 'programatically' in job.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064264#comment-14064264
]
Hadoop QA commented on MAPREDUCE-5972:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656108/MAPREDUCE-5972.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-tools/hadoop-sls:
org.apache.hadoop.ipc.TestIPC
org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4743//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4743//console
This message is automatically generated.
Fix typo 'programatically' in job.xml
-
Key: MAPREDUCE-5972
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5972
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5972.patch
In job.xml, there's a typo 'programatically' as the below if a property is
set through program.
{code}
property
namemapreduce.job.map.class/name
valueorg.apache.hadoop.examples.WordCount$TokenizerMapper/value
sourceprogramatically/source
/property
{code}
should be 'programmatically'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5971) Move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated MAPREDUCE-5971: --- Summary: Move the default options for distcp -p to DistCpOptionSwitch (was: move the default options for distcp -p to DistCpOptionSwitch) Move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5971) Move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated MAPREDUCE-5971: --- Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2, thanks Charles Move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Fix For: 2.6.0 Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-5933: Assignee: Robert Kanter (was: Zhijie Shen) Enable MR AM to post history events to the timeline server -- Key: MAPREDUCE-5933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am Reporter: Zhijie Shen Assignee: Robert Kanter Nowadays, MR AM collects the history events and writes it to HDFS for JHS to source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Status: Open (was: Patch Available)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Attachment: MAPREDUCE-5957.patch
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Status: Patch Available (was: Open)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5971) Move the default options for distcp -p to DistCpOptionSwitch
[ https://issues.apache.org/jira/browse/MAPREDUCE-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064396#comment-14064396 ] Hudson commented on MAPREDUCE-5971: --- FAILURE: Integrated in Hadoop-trunk-Commit #5895 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5895/]) MAPREDUCE-5971. Move the default options for distcp -p to DistCpOptionSwitch. Contributed by Charles Lamb. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611217) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java Move the default options for distcp -p to DistCpOptionSwitch Key: MAPREDUCE-5971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: trunk Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Fix For: 2.6.0 Attachments: MAPREDUCE-5971.001.patch, MAPREDUCE-5971.002.patch The default preserve flags for distcp -p are embedded in the OptionsParser code. Refactor to co-locate them with the actual flag initialization. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064402#comment-14064402
]
Hadoop QA commented on MAPREDUCE-5957:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656184/MAPREDUCE-5957.patch
against trunk revision .
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4745//console
This message is automatically generated.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
MAPREDUCE-5957.patch, MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
