[jira] [Created] (MAPREDUCE-5812) Make task context available to OutputCommitter.isRecoverySupported()
Mohammad Kamrul Islam created MAPREDUCE-5812: Summary: Make task context available to OutputCommitter.isRecoverySupported() Key: MAPREDUCE-5812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5812 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Background == The system like Hive provides its version of OutputCommitter. The custom implementation of isRecoverySupported() requires task context. From taskContext:getConfiguration(), hive checks if hive-defined specific property is set or not. Based on the property value, it returns true or false. However, in the current OutputCommitter:isRecoverySupported(), there is no way of getting task config. As a result, user can't turn on/off the MRAM recovery feature. Proposed resolution: === 1. Pass Task Context into isRecoverySupported() method. Pros: Easy and clean Cons: Possible backward compatibility issue due to aPI changes. (Is it true?) 2. Call outputCommitter.setupTask(taskContext) from MRAM: The new OutputCommitter will store the context in the class level variable and use it from isRecoverySupported() Props: No API changes. No backward compatibility issue. This call can be made from MRAppMaster.getOutputCommitter() method for old API case. Cons: Might not be very clean solution due to class level variable. Please give your comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4957) Throw FileNotFoundException when running in single node and mapreduce.framework.name is local
[ https://issues.apache.org/jira/browse/MAPREDUCE-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated MAPREDUCE-4957: -- Assignee: (was: Yi Liu) Throw FileNotFoundException when running in single node and mapreduce.framework.name is local --- Key: MAPREDUCE-4957 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4957 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yi Liu Priority: Minor Attachments: MAPREDUCE-4957.patch, MAPREDUCE-4957.patch Run in single node and mapreduce.framework.name is local, and get following error: java.io.FileNotFoundException: File does not exist: /root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:772) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:292) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:365) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:446) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar)' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (MAPREDUCE-4957) Throw FileNotFoundException when running in single node and mapreduce.framework.name is local
[ https://issues.apache.org/jira/browse/MAPREDUCE-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu reassigned MAPREDUCE-4957: - Assignee: Yi Liu Throw FileNotFoundException when running in single node and mapreduce.framework.name is local --- Key: MAPREDUCE-4957 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4957 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yi Liu Assignee: Yi Liu Priority: Minor Attachments: MAPREDUCE-4957.patch, MAPREDUCE-4957.patch Run in single node and mapreduce.framework.name is local, and get following error: java.io.FileNotFoundException: File does not exist: /root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:772) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:254) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:292) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:365) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:446) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar)' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4957) Throw FileNotFoundException when running in single node and mapreduce.framework.name is local
[
https://issues.apache.org/jira/browse/MAPREDUCE-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949046#comment-13949046
]
Hadoop QA commented on MAPREDUCE-4957:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566460/MAPREDUCE-4957.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4462//console
This message is automatically generated.
Throw FileNotFoundException when running in single node and
mapreduce.framework.name is local
---
Key: MAPREDUCE-4957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4957
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
Attachments: MAPREDUCE-4957.patch, MAPREDUCE-4957.patch
Run in single node and mapreduce.framework.name is local, and get following
error:
java.io.FileNotFoundException: File does not exist:
/root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:772)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:254)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:292)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:365)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1450)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:446)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.FileNotFoundException(File does
not exist:
/root/proj/hive-trunk/build/dist/lib/hive-builtins-0.11.0-SNAPSHOT.jar)'
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949136#comment-13949136 ] Madhurima commented on MAPREDUCE-2911: -- Hi, iam interested in Hamster and want to build and test the sources. Please let me know f the sources of Hamster are available for build and testing. thanks, Madhurima Hamster: Hadoop And Mpi on the same cluSTER --- Key: MAPREDUCE-2911 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Environment: All Unix-Environments Reporter: Milind Bhandarkar Original Estimate: 336h Remaining Estimate: 336h MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949168#comment-13949168
]
Hudson commented on MAPREDUCE-5805:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk #522 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/522/])
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
hyphens and thus avoid parsing errors. Contributed by Akira Ajisaka. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1582125)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
Unable to parse launch time from job history file
-
Key: MAPREDUCE-5805
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Fengdong Yu
Assignee: Akira AJISAKA
Fix For: 2.4.0
Attachments: MAPREDUCE-5805.patch
when job complete, there are WARN complains in the log:
{code}
2014-03-19 13:31:10,036 WARN
org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
launch time from job history file
job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
: java.lang.NumberFormatException: For input string: queue
{code}
because there is (-) in the queue name 'test-queue', we split the job
history file name by (-), and get the ninth item as job start time.
FileNameIndexUtils.java
{code}
private static final int JOB_START_TIME_INDEX = 9;
{code}
but there is another potential issue:
if I also include '-' in the job name(test_one_world in this case), there are
all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
Gera Shegalov created MAPREDUCE-5813:
Summary: YarnChild does not load job.xml with
mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Priority: Blocker
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default class
loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to read
the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov reassigned MAPREDUCE-5813:
Assignee: Gera Shegalov
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5813:
-
Attachment: MAPREDUCE-5813.v01.patch
v01 patch for review.
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Gera Shegalov created MAPREDUCE-5814:
Summary: fat jar with *-default.xml may fail when
mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more recent
guava as a dependency. The job failed because the cluster's {{*-default.xml}}
files were overshadowed by the ones in the fat jar. We propose to treat these
default config files like the system packages {{org.apache.hadoop.}} to avoid a
counterintuitivie behavior as if we had {{mapreduce.job.user.classpath.first}}
set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5814:
-
Attachment: MAPREDUCE-5814.v01.patch
v01 patch for review.
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5813:
-
Status: Patch Available (was: Open)
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5814:
-
Assignee: Gera Shegalov
Status: Patch Available (was: Open)
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949258#comment-13949258
]
Rohith commented on MAPREDUCE-5397:
---
[~jianhe], I am using Hadoop-2.3 released version. I am facing same problem
again and again at least once in day. All application attempts failed to start
HTTP server in tern NPE while getting webApp.getHttpPort() at registeration.
With same cluster and client configurations I submitted new job and got
succeeded. I couldn't get real cause for the issue, but observation is
1. After job is failed, I checked for directory
*/tmp/Jetty_0_0_0_0_59894_mapreducen26zu/webapp* in container run. I could
not see this directory, is it bug in Jetty!!?
2. All attempt for this job failed, but with same client configuration new job
got succeeded.
AM crashes because Webapp failed to start on multi node cluster
---
Key: MAPREDUCE-5397
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: log.txt
I set up a 12 nodes cluster and tried submitting jobs but get this exception.
But job is able to succeed after AM crashes and retry a few times(2 or 3)
{code}
2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of
context
org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
java.io.FileNotFoundException:
/tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at java.io.FileOutputStream.init(FileOutputStream.java:145)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at
org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at
org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at
org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949288#comment-13949288
]
Hadoop QA commented on MAPREDUCE-5813:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637128/MAPREDUCE-5813.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4464//console
This message is automatically generated.
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949280#comment-13949280
]
Hadoop QA commented on MAPREDUCE-5814:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637135/MAPREDUCE-5814.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4463//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4463//console
This message is automatically generated.
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949320#comment-13949320
]
Hudson commented on MAPREDUCE-5805:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1739 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1739/])
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
hyphens and thus avoid parsing errors. Contributed by Akira Ajisaka. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1582125)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
Unable to parse launch time from job history file
-
Key: MAPREDUCE-5805
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Fengdong Yu
Assignee: Akira AJISAKA
Fix For: 2.4.0
Attachments: MAPREDUCE-5805.patch
when job complete, there are WARN complains in the log:
{code}
2014-03-19 13:31:10,036 WARN
org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
launch time from job history file
job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
: java.lang.NumberFormatException: For input string: queue
{code}
because there is (-) in the queue name 'test-queue', we split the job
history file name by (-), and get the ninth item as job start time.
FileNameIndexUtils.java
{code}
private static final int JOB_START_TIME_INDEX = 9;
{code}
but there is another potential issue:
if I also include '-' in the job name(test_one_world in this case), there are
all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949353#comment-13949353
]
Hudson commented on MAPREDUCE-5805:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1714 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1714/])
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
hyphens and thus avoid parsing errors. Contributed by Akira Ajisaka. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1582125)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
Unable to parse launch time from job history file
-
Key: MAPREDUCE-5805
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Fengdong Yu
Assignee: Akira AJISAKA
Fix For: 2.4.0
Attachments: MAPREDUCE-5805.patch
when job complete, there are WARN complains in the log:
{code}
2014-03-19 13:31:10,036 WARN
org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
launch time from job history file
job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
: java.lang.NumberFormatException: For input string: queue
{code}
because there is (-) in the queue name 'test-queue', we split the job
history file name by (-), and get the ninth item as job start time.
FileNameIndexUtils.java
{code}
private static final int JOB_START_TIME_INDEX = 9;
{code}
but there is another potential issue:
if I also include '-' in the job name(test_one_world in this case), there are
all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949654#comment-13949654
]
Jian He commented on MAPREDUCE-5397:
My impression on this issue was I submitted a job, the first few attempts(2 or
3) of the job all failed because of the above reason. Eventually the last
attempt got passed. But after I made a clean build and re-deploy the cluster,
I couldn't reproduce anymore. Feel free to reopen this if necessary, and also
share some logs. tx
AM crashes because Webapp failed to start on multi node cluster
---
Key: MAPREDUCE-5397
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: log.txt
I set up a 12 nodes cluster and tried submitting jobs but get this exception.
But job is able to succeed after AM crashes and retry a few times(2 or 3)
{code}
2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of
context
org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
java.io.FileNotFoundException:
/tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at java.io.FileOutputStream.init(FileOutputStream.java:145)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at
org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at
org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at
org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949751#comment-13949751
]
Sangjin Lee commented on MAPREDUCE-5814:
How about *-site.xml files? Do we support clients sending their site.xml files
as part of app submission and overriding the cluster's site.xml files?
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949774#comment-13949774
]
Gera Shegalov commented on MAPREDUCE-5814:
--
Sangjin, I was thinking of this... but *-site.xml unlike job.xml are not meant
to be shipped, they are meant to be installed. default.xml is packaged with
hadoop, and can therefore make it into a fat jar. It's a good idea as a
defensive measure. The only reason that prevented me from doing this is that
the list of such files can be quite long, and include all kinds of log4j and
other properties. But they can be legally shipped too. So far I have settled
for the least requirement. Looking forward to the pros and cons.
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5607) Backport MAPREDUCE-5086 - MR app master deletes staging dir when sent a reboot command from the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5607: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) This feature change is introduces too much risk to so close to the end of 0.23.x development and the beginning of maintenance for this line. Backport MAPREDUCE-5086 - MR app master deletes staging dir when sent a reboot command from the RM -- Key: MAPREDUCE-5607 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5607 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-5607-branch-0.23.patch If the RM is restarted when the MR job is running, then it sends a reboot command to the job. The job ends up deleting the staging dir and that causes the next attempt to fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4941) Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
[
https://issues.apache.org/jira/browse/MAPREDUCE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949938#comment-13949938
]
Jonathan Eagles commented on MAPREDUCE-4941:
[~jlowe], this patch no longer applies. Can you update when you get a chance?
Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
Key: MAPREDUCE-4941
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4941
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Minor
Attachments: MAPREDUCE-4941.patch, MAPREDUCE-4941.patch
Unlike its counterpart in org.apache.hadoop.mapreduce.lib.input, the
CombineFileRecordReader in mapred requires a user to cast to a RecordReader
since the constructor specification says it must have the RecordReaderK,V
class as a parameter. It should use {{Class? extends RecordReaderK,V}}
like its mapreduce counterpart to make it easier to use.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-4941) Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
[
https://issues.apache.org/jira/browse/MAPREDUCE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949953#comment-13949953
]
Hadoop QA commented on MAPREDUCE-4941:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578762/MAPREDUCE-4941.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4465//console
This message is automatically generated.
Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
Key: MAPREDUCE-4941
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4941
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Minor
Attachments: MAPREDUCE-4941.patch, MAPREDUCE-4941.patch
Unlike its counterpart in org.apache.hadoop.mapreduce.lib.input, the
CombineFileRecordReader in mapred requires a user to cast to a RecordReader
since the constructor specification says it must have the RecordReaderK,V
class as a parameter. It should use {{Class? extends RecordReaderK,V}}
like its mapreduce counterpart to make it easier to use.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
Gera Shegalov created MAPREDUCE-5815: Summary: Fix NPE in TestMRAppMaster Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5815: - Attachment: MAPREDUCE-5815.v01.patch v01 patch for review Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5815: - Status: Patch Available (was: Open) Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950259#comment-13950259
]
Gera Shegalov commented on MAPREDUCE-5813:
--
The TestMRAppMaster NPE are related to MAPREDUCE-5805. This JIRA touches only
YarnChild for Map/ReduceTask
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950285#comment-13950285
]
Hadoop QA commented on MAPREDUCE-5815:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637306/MAPREDUCE-5815.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//console
This message is automatically generated.
Fix NPE in TestMRAppMaster
--
Key: MAPREDUCE-5815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mrv2
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5815.v01.patch
Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to
be introduced by MAPREDUCE-5805.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950315#comment-13950315
]
Sangjin Lee commented on MAPREDUCE-5813:
The patch looks good to me. The only nit is to add the right job conf before
Limits.init() so that Limits.init() can use the right job conf.
It's nearly the same fix, but how about
{code}
final JobConf job = new JobConf(MRJobConfig.JOB_CONF_FILE);
Limits.init(job);
...
{code}
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950331#comment-13950331
]
Sangjin Lee commented on MAPREDUCE-5814:
Sounds good. I agree users adding *-site.xml files in their jar is such an edge
case that I don't think we need to be defensive about that type of scenarios as
well. The *-default.xml files can definitely get in via build, but not
*-site.xml...
fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Key: MAPREDUCE-5814
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5814.v01.patch
We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had
to run with {{mapreduce.job.classloader=true}} because it needed a more
recent guava as a dependency. The job failed because the cluster's
{{*-default.xml}} files were overshadowed by the ones in the fat jar. We
propose to treat these default config files like the system packages
{{org.apache.hadoop.}} to avoid a counterintuitivie behavior as if we had
{{mapreduce.job.user.classpath.first}} set.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950414#comment-13950414
]
Rohith commented on MAPREDUCE-5397:
---
I reopen this issue since I am facing this at least once in a day randomly.
AM crashes because Webapp failed to start on multi node cluster
---
Key: MAPREDUCE-5397
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: log.txt
I set up a 12 nodes cluster and tried submitting jobs but get this exception.
But job is able to succeed after AM crashes and retry a few times(2 or 3)
{code}
2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of
context
org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
java.io.FileNotFoundException:
/tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at java.io.FileOutputStream.init(FileOutputStream.java:145)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at
org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at
org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at
org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
