Sandy:
Yes. Attached is the segment from the job tracker log file that
shows the error and stack traces.
The maven surefire report for the test shows an assertion failure on the
following line from
the test:
assertTrue(job.run(true));
-------------------------------------------------------------------------------
Test set: org.apache.giraph.io.TestJsonBase64Format
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.363
sec <<< FAILURE!
testContinue(org.apache.giraph.io.TestJsonBase64Format) Time elapsed:
32.352 sec <<< FAILURE!
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at
org.apache.giraph.io.TestJsonBase64Format.testContinue(TestJsonBase64Format.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
at
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)
at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at
org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Below is the surefire report stack trace:
On 2/25/2013 6:55 PM, Sandy Ryza wrote:
Great to hear it helped. Are you able to provide the full stack trace
for that exception?
thanks,
Sandy
On Mon, Feb 25, 2013 at 3:51 PM, David Boyd
<[email protected] <mailto:[email protected]>> wrote:
Sandy:
Thanks that helps a great deal. I am now at least getting to
the point that the jobs show up in the job tracker. However,
they all
fail on initialization with the good old:
java.io.FileNotFoundException: File
/tmp/hadoop-mapred/mapred/staging/hdfs/.staging/job_201302211213_0055/job.jar
does not exist
This tells me that maven is either not specifying that the
giraph-core jar file should be used as the job jar or I am missing
something else in the set up.
Attached is the job.xml file from one of the failed jobs and below
is the relevant profile out of my pom.xml.
I did upgrade to CDH4.1.3 just to see if that would help.
Also, I have been running all sorts of jobs (benchmarks, and other
tests) against this cluster for some time so I know that the cluster
works well.
Again, any help is appreciated.
Relevant section of pom.xml:
<profile>
<id>hadoop_cdh4.1.3mr1</id>
<properties>
<hadoopmr1.version>2.0.0-mr1-cdh4.1.3</hadoopmr1.version>
<hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
<munge.symbols>HADOOP_1_SECURITY,HADOOP_1_SECRET_MANAGER</munge.symbols>
</properties>
<dependencies>
<!-- sorted lexicographically -->
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoopmr1.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<version>${hadoopmr1.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
</profile>
On 2/25/2013 12:47 PM, Sandy Ryza wrote:
Hi David,
Moving this to cdh-user, as it is CDH-specific.
CDH4 comes with two versions of mapreduce, MR1, and MR2. It
sounds like you are building against MR2
(http://blog.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/).
Do you know whether your cluster runs MR2/YARN or MR1? If it
runs, MR2, you can set mapreduce.framework.name
<http://mapreduce.framework.name> to "yarn". If it runs MR1, you
can build against the MR1 jar by setting the version of your
hadoop-client to 2.0.0-mr1-cdh4.1.1.
(https://ccp.cloudera.com/display/CDH4DOC/Managing+Hadoop+API+Dependencies+in+CDH4)
Does that help?
-Sandy
On Mon, Feb 25, 2013 at 8:26 AM, David Boyd
<[email protected]
<mailto:[email protected]>> wrote:
All:
I am trying to get the Giraph 0.2 snapshot (pulled via GIT
on Friday)
to build and run with CDH4.
I modified the pom.xml to provide a profile for my specific
version (4.1.1).
The build works (mvn -Phadoop_cdh4.1.1 clean package test)
and passes
all the tests.
If I try to do the next step and submit to my cluster with
the command:
mvn -Phadoop_cdh4.1.1 test
-Dprop.mapred.job.tracker=10.1.94.53:8021
<http://10.1.94.53:8021> -Dgiraph.zkList=10.1.94.104:2181
<http://10.1.94.104:2181>
the JSON test in core fails. If I move that test out of the
way a whole bunch of tests in examples
fail. They all fail with:
java.io.IOException: Cannot initialize Cluster. Please
check your
configuration for mapreduce.framework.name
<http://mapreduce.framework.name> and the correspond server
addresses.
I have tried passing mapreduce.framework.name
<http://mapreduce.framework.name> as both local and classic.
I have also set those values in my mapreduce-site.xml.
Interestingly I can run the pagerank benchmark in code with
the command:
hadoop jar
./giraph-core/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-cdh4.1.3-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark
-Dmapred.child.java-opts="-Xmx64g -Xms64g
XX:+UseConcMarkSweepGC
-XX:-UseGCOverheadLimit" -Dgiraph.zkList=10.1.94.104:2181
<http://10.1.94.104:2181> -e 1 -s 3 -v
-V 50000 -w 83
And it completes just fine.
I have searched high and low for documents and examples on
how to run the example programs from other
than maven but have not found any thing.
Any help or suggestions would be greatly appreciated.
THanks.
--
========= mailto:[email protected]
<mailto:[email protected]> ============
David W. Boyd
Director, Engineering, Research and Development
Data Tactics Corporation
7901 Jones Branch, Suite 240
Mclean, VA 22102
office: +1-703-506-3735, ext 308
<tel:%2B1-703-506-3735%2C%20ext%20308>
fax: +1-703-506-6703 <tel:%2B1-703-506-6703>
cell: +1-703-402-7908 <tel:%2B1-703-402-7908>
============== http://www.data-tactics.com/ ============
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
--
=========mailto:[email protected] ============
David W. Boyd
Director, Engineering, Research and Development
Data Tactics Corporation
7901 Jones Branch, Suite 240
Mclean, VA 22102
office:+1-703-506-3735, ext 308 <tel:%2B1-703-506-3735%2C%20ext%20308>
fax:+1-703-506-6703 <tel:%2B1-703-506-6703>
cell:+1-703-402-7908 <tel:%2B1-703-402-7908>
==============http://www.data-tactics.com/ ============
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
--
========= mailto:[email protected] ============
David W. Boyd
Director, Engineering, Research and Development
Data Tactics Corporation
7901 Jones Branch, Suite 240
Mclean, VA 22102
office: +1-703-506-3735, ext 308
fax: +1-703-506-6703
cell: +1-703-402-7908
============== http://www.data-tactics.com/ ============
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
2013-02-25 23:40:51,446 INFO org.apache.hadoop.mapred.JobTracker: Initializing
job_201302211213_0056
2013-02-25 23:40:51,447 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201302211213_0056
2013-02-25 23:40:51,454 INFO org.apache.hadoop.mapred.JobTracker: Job
job_201302211213_0056 added successfully for user 'hdfs' to queue 'default'
2013-02-25 23:40:51,491 INFO org.apache.hadoop.mapred.JobInProgress: jobToken
generated and stored with users keys in
/tmp/hadoop-mapred/mapred/system/job_201302211213_0056/jobToken
2013-02-25 23:40:51,495 INFO org.apache.hadoop.mapred.JobInProgress: Input size
for job job_201302211213_0056 = 0. Number of splits = 4
2013-02-25 23:40:51,495 INFO org.apache.hadoop.mapred.JobInProgress:
job_201302211213_0056 LOCALITY_WAIT_FACTOR=0.0
2013-02-25 23:40:51,496 INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201302211213_0056 initialized successfully with 4 map tasks and 0 reduce
tasks.
2013-02-25 23:40:51,625 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-25 23:40:51,625 INFO org.apache.hadoop.mapred.JobTracker: Adding task
(JOB_SETUP) 'attempt_201302211213_0056_m_000005_0' to tip
task_201302211213_0056_m_000005, for tracker
'tracker_r102u12.xdata.data-tactics-corp.com:localhost/127.0.0.1:53819'
2013-02-25 23:40:51,928 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from attempt_201302211213_0056_m_000005_0: Error initializing
attempt_201302211213_0056_m_000005_0:
java.io.FileNotFoundException: File
/tmp/hadoop-mapred/mapred/staging/hdfs/.staging/job_201302211213_0056/job.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:455)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:364)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:231)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:183)
at
org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:87)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1897)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:231)
at org.apache.hadoop.mapred.TaskTracker$4.run(Task
2013-02-25 23:40:51,929 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to
set finish time for task attempt_201302211213_0056_m_000005_0 when no start
time is set, stackTrace is : java.lang.Exception
at
org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:185)
at
org.apache.hadoop.mapred.TaskInProgress.incompleteSubTask(TaskInProgress.java:670)
at
org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2880)
at
org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1165)
at
org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4278)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3216)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2013-02-25 23:40:51,929 INFO org.apache.hadoop.mapred.TaskInProgress:
TaskInProgress task_201302211213_0056_m_000005 has failed 1 times.
2013-02-25 23:40:51,929 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.JobInProgress$Counter is deprecated. Use
org.apache.hadoop.mapreduce.JobCounter instead
2013-02-25 23:40:51,929 INFO org.apache.hadoop.mapred.JobInProgress: Aborting
job job_201302211213_0056
2013-02-25 23:40:51,929 INFO org.apache.hadoop.mapred.JobInProgress: Killing
job 'job_201302211213_0056'
2013-02-25 23:40:51,929 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-25 23:40:51,929 INFO org.apache.hadoop.mapred.JobTracker: Adding task
(JOB_CLEANUP) 'attempt_201302211213_0056_m_000004_0' to tip
task_201302211213_0056_m_000004, for tracker
'tracker_r102u12.xdata.data-tactics-corp.com:localhost/127.0.0.1:53819'
2013-02-25 23:40:51,930 INFO org.apache.hadoop.mapred.JobTracker: Removing task
'attempt_201302211213_0056_m_000005_0'
2013-02-25 23:40:52,233 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from attempt_201302211213_0056_m_000004_0: Error initializing
attempt_201302211213_0056_m_000004_0:
java.io.FileNotFoundException: File
/tmp/hadoop-mapred/mapred/staging/hdfs/.staging/job_201302211213_0056/job.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:455)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:364)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:231)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:183)
at
org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:87)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1897)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:231)
at org.apache.hadoop.mapred.TaskTracker$4.run(Task
2013-02-25 23:40:52,233 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to
set finish time for task attempt_201302211213_0056_m_000004_0 when no start
time is set, stackTrace is : java.lang.Exception
at
org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:185)
at
org.apache.hadoop.mapred.TaskInProgress.incompleteSubTask(TaskInProgress.java:670)
at
org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2880)
at
org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1165)
at
org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4278)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3216)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2013-02-25 23:40:52,233 INFO org.apache.hadoop.mapred.TaskInProgress:
TaskInProgress task_201302211213_0056_m_000004 has failed 1 times.
2013-02-25 23:40:52,234 INFO org.apache.hadoop.mapred.JobInProgress: Aborting
job job_201302211213_0056
2013-02-25 23:40:52,234 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary:
jobId=job_201302211213_0056,submitTime=1361835651353,launchTime=1361835651495,firstJobSetupTaskLaunchTime=1361835651625,firstJobCleanupTaskLaunchTime=1361835651929,finishTime=1361835652234,numMaps=4,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=hdfs,queue=default,status=FAILED,mapSlotSeconds=0,reduceSlotsSeconds=0,clusterMapCapacity=84,clusterReduceCapacity=56
2013-02-25 23:40:52,235 INFO org.apache.hadoop.mapred.JobHistory: Moving
file:/var/log/hadoop-0.20-mapreduce/history/job_201302211213_0056_1361835651353_hdfs_testContinue
to
file:/var/log/hadoop-0.20-mapreduce/history/done/xd-gp-jt.xdata.data-tactics-corp.com_1361448787061_/2013/01/25/000000
2013-02-25 23:40:52,235 INFO org.apache.hadoop.mapred.JobTracker: Removing task
'attempt_201302211213_0056_m_000004_0'
2013-02-25 23:40:52,237 INFO org.apache.hadoop.mapred.JobHistory: Moving
file:/var/log/hadoop-0.20-mapreduce/history/xd-gp-jt.xdata.data-tactics-corp.com_1361448787061_job_201302211213_0056_conf.xml
to
file:/var/log/hadoop-0.20-mapreduce/history/done/xd-gp-jt.xdata.data-tactics-corp.com_1361448787061_/2013/01/25/000000
2013-02-25 23:43:28,059 INFO org.apache.hadoop.conf.Configuration: found
resource webapps/static/jobconf.xsl at
file:/usr/lib/hadoop-0.20-mapreduce/webapps/static/jobconf.xsl
2013-02-25 23:48:07,047 INFO org.apache.hadoop.conf.Configuration: found
resource webapps/static/jobconf.xsl at
file:/usr/lib/hadoop-0.20-mapreduce/webapps/static/jobconf.xsl
2013-02-25 23:48:21,573 INFO org.apache.hadoop.conf.Configuration: found
resource webapps/static/jobconf.xsl at
file:/usr/lib/hadoop-0.20-mapreduce/webapps/static/jobconf.xsl
