Hello, so I'm watching this thread with interest as well, since that's
the exact example I'll be trying to run as well, and I see Silvio
appears to be asking about running Giraph 1.0 against Hadoop 2.0.0.
I'm curious, is that a known working combination, 1.0 vs 2.0.x? And if
so, are there patches to apply first and what maven command line should
I build with? (ie. will a -Dhadoop.version=2.0.5 give the needed MR
based API or will it use YARN?) I figure it's better to ask than
aimlessly try all the combinations. :)
Thanks!
On 13-10-07 11:01 AM, Claudio Martella wrote:
try to increase the mappers' heap space through mapred.child.java.opts.
On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
<[email protected] <mailto:[email protected]>> wrote:
I think now my issue is another, thanks to all.
java.lang.IllegalStateException: run: Caught an unrecoverable exception
waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
......
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError:*Java heap space*
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
... 13 more
2013/10/7 Claudio Martella <[email protected]
<mailto:[email protected]>>
OK, thanks.
I really have to push that patch in.
On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
<[email protected]
<mailto:[email protected]>> wrote:
yes i do,
i have seen this in your post in:
http://www.mail-archive.com/[email protected]/msg00957.html
excuse me if i had checked in the mail-achive first I would
have avoided the last post.
Now zk issue are resolved.
2013/10/7 Claudio Martella <[email protected]
<mailto:[email protected]>>
Can you try applying this one first?
http://www.mail-archive.com/[email protected]/msg00945/check.diff
On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
<[email protected]
<mailto:[email protected]>> wrote:
*As i said i have builded*
/giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar/
*for cdh4, successfully. The job start to monitoring the
success rate:*
/13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%/
*but then*
/Error running child
java.lang.IllegalStateException: run: Caught an
unrecoverable exception java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
/
/.../
/Caused by: java.lang.RuntimeException:
java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist./
2013/10/5 Silvio Di gregorio
<[email protected]
<mailto:[email protected]>>
I ha ve build w/ hadoop_cdh4.1.2 parameter.
Something is changed, monday i report the
result. Now the farm is closed.
Il giorno 05/ott/2013 14:06, "Claudio Martella"
<[email protected]
<mailto:[email protected]>> ha scritto:
Oh, right, -vof is in trunk. Anyway it looks
like you built giraph for the wrong profile.
You mentioned you're running on 2.0, but
your giraph is built for 0.20.203
<tel:0.20.203>. try building with a profile
for your hadoop version.
On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di
gregorio <[email protected]
<mailto:[email protected]>> wrote:
org.apache.commons.cli.UnrecognizedOptionException:
Unrecognized option: -vof
in 1.0 version is
-of,--outputFormat <arg>
Vertex output format
-op,--outputPath <arg>
Vertex output path
2013/10/4 Claudio Martella
<[email protected]
<mailto:[email protected]>>
did you try the argument (-vof) i
suggested?
On Fri, Oct 4, 2013 at 2:13 PM,
Silvio Di gregorio
<[email protected]
<mailto:[email protected]>> wrote:
i've specified -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat
but the same error was produced
Exception in thread "main"
java.lang.IncompatibleClassChangeError:
Found interface
org.apache.hadoop.mapreduce.JobContext,
but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
at
java.security.AccessController.doPrivileged(Native
Method)
at
javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
at
org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
at
org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
at
org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.RunJar.main(RunJar.java:208)
2013/10/4 Claudio Martella
<[email protected]
<mailto:[email protected]>>
Hi,
you need to specify the
vertex outputformat class
(-vof option), e.g.
org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
On Fri, Oct 4, 2013 at 1:06
PM, Silvio Di gregorio
<[email protected]
<mailto:[email protected]>>
wrote:
Hi,
I hope I have sent to
the right address.
i have a graph (directed
and unweighted) stored
in hdfs like a adjacency
list (140Milions of
edges 6Milions of vertex)
node<tab>neighbors
23 2 1 3 4 3
1 999 99923
<tel:1%20999%2099923>
909 .......
......
hadoop version Hadoop
2.0.0-cdh4.3.0 - java 1.6
I have executed the
giraph-1.0 connected
components example, in
this fashion
> hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0
<tel:0.20.203.0>-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.ConnectedComponentsVertex
-vif
org.apache.giraph.io.formats.IntIntNullTextInputFormat
-vip
/user/hdfs/lista_adj_txt
-op connectedgiraph
--workers 4
and then fail with:
13/10/04 09:28:29 INFO
utils.ConfigurationUtils: No
edge input format
specified. Ensure your
InputFormat does not
require one.
13/10/04 09:28:29 INFO
utils.ConfigurationUtils: No
output format specified.
Ensure your OutputFormat
does not require one.
13/10/04 09:28:30 INFO
job.GiraphJob: run:
Since checkpointing is
disabled (default), do
not allow any task
retries (setting
mapred.map.max.attempts
= 0, old value = 4)
13/10/04 09:28:31 WARN
mapred.JobClient: Use
GenericOptionsParser for
parsing the arguments.
Applications should
implement Tool for the same.
13/10/04 09:28:31 INFO
mapred.JobClient:
Cleaning up the staging
area
hdfs://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
Exception in thread
"main"
java.lang.IncompatibleClassChangeError:
Found interface
org.apache.hadoop.mapreduce.JobContext,
but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
......
Thanks in advance
--
Claudio Martella
[email protected]
<mailto:[email protected]>
--
Claudio Martella
[email protected]
<mailto:[email protected]>
--
Claudio Martella
[email protected]
<mailto:[email protected]>
--
Claudio Martella
[email protected]
<mailto:[email protected]>
--
Claudio Martella
[email protected] <mailto:[email protected]>
--
Claudio Martella
[email protected] <mailto:[email protected]>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
