Here's an easy way to calculate the number of map slots required.
workers + 1. So for instance if you only have 2 map slots, you can only
run at most one worker. If you're running on your laptop, feel free to
change the number of map slots to have you need.
Hope that helps,
Avery
On 5/7/13 11:40 AM, Kiru Pakkirisamy wrote:
Avery et al,
Please let me know if I should use Giraph only on a multi-node cluster
(what is the min # of nodes ?)
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
--- On *Mon, 5/6/13, Kiru Pakkirisamy /<[email protected]>/*
wrote:
From: Kiru Pakkirisamy <[email protected]>
Subject: TestJsonBase64Format failure on 1.0.0
To: [email protected]
Date: Monday, May 6, 2013, 9:40 AM
I got over my compilation issues (thanks - @Avery, @Roman).
Now, I am trying to run the test and one pearticular test is failing.
I want to get to the bottom of this, because I am unable to run
the PageRank example. Maybe, it is because I have only on
tasktracker (?) (Apache pseudo-cluster on my ubuntu laptop).
Regards,
- kiru
2013-05-06 09:12:32,197 INFO org.apache.hadoop.mapred.JobTracker:
Adding task (MAP) 'attempt_201305052325_0013_m_000003_0' to tip
task_201305052325_0013_m_000003, for tracker
'tracker_kiru-N53SV:localhost/127.0.0.1:42265'
2013-05-06 09:12:35,198 INFO
org.apache.hadoop.mapred.TaskInProgress: Error from
attempt_201305052325_0013_m_000002_0:
java.lang.IllegalStateException: run: Caught an unrecoverable
exception waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.IllegalStateException: waitFor:
ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
at
org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111)
at
org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73)
at
org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192)
at
org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:276)
at
org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:323)
at
org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506)
at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:230)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: call: IOException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at
org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271)
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
... 15 more
Caused by: java.lang.IllegalStateException: call: IOException
at
org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:172)
at
org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:58)
at
org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException:
/tmp/_giraphTests/testContinue/_logs (Is a directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>(RawLocalFileSystem.java:71)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:107)
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:177)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:126)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
at
org.apache.giraph.io.formats.TextVertexInputFormat$TextVertexReader.initialize(TextVertexInputFormat.java:96)
at
org.apache.giraph.io.formats.JsonBase64VertexInputFormat$JsonBase64VertexReader.initialize(JsonBase64VertexInputFormat.java:71)
at
org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:120)
at
org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:220)
at
org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:161)
... 7 more
2013-05-06 09:22:44,485 INFO
org.apache.hadoop.mapred.TaskInProgress: Error from
attempt_201305052325_0013_m_000002_0: Task
attempt_201305052325_0013_m_000002_0 failed to report status for
602 seconds. Killing!
2013-05-06 09:22:44,485 INFO
org.apache.hadoop.mapred.TaskInProgress: TaskInProgress
task_201305052325_0013_m_000002 has failed 1 times.
2013-05-06 09:22:44,485 INFO
org.apache.hadoop.mapred.JobInProgress: Aborting job
job_201305052325_0013
2013-05-06 09:22:44,485 INFO
org.apache.hadoop.mapred.JobInProgress: Killing job
'job_201305052325_0013'
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
--- On *Mon, 5/6/13, Kiru Pakkirisamy
/<[email protected]>/*wrote:
From: Kiru Pakkirisamy <[email protected]>
Subject: Re: Compiling 1.0.0 distribution
To: [email protected], "Avery Ching" <[email protected]>
Date: Monday, May 6, 2013, 12:02 AM
Yes, I am trying to run on my Ubuntu laptop. Let me look at
the log files. Thanks for the help. Much appreciated.
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
--- On *Sun, 5/5/13, Avery Ching /<[email protected]>/* wrote:
From: Avery Ching <[email protected]>
Subject: Re: Compiling 1.0.0 distribution
To: [email protected]
Cc: "Kiru Pakkirisamy" <[email protected]>
Date: Sunday, May 5, 2013, 11:51 PM
My guess is that you don't have enough workers to run the
job and the master kills the job (i.e. are you running on
a single machine setup?). You can try to run first with
one worker (this will take 2 map slots - one for the
master and one for the worker). You can also look at the
logs from map task 0 to see more clearly what the error was.
Avery
On 5/5/13 11:16 PM, Kiru Pakkirisamy wrote:
Yup, I did a mvn3 install and then a mvn3 compile to get
around that already.
Right now, I am trying to run the PageRank, even after a
few runs I have not had one successful run . The maps
progress decreases in percentage (second time around) !!
I have never seen this before (?)
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
--- On *Sun, 5/5/13, Roman Shaposhnik /<[email protected]>/*
wrote:
From: Roman Shaposhnik <[email protected]>
Subject: Re: Compiling 1.0.0 distribution
To: [email protected]
Date: Sunday, May 5, 2013, 10:50 PM
To pile on top of that -- you can also run mvn -pl
module-name from the top
level to short-circuit the build to that module (and
yet still honor the dependencies).
Thanks,
Roman.
On Sun, May 5, 2013 at 10:44 PM, Avery Ching
<[email protected]> wrote:
The easiest way is to compile from the base
directory, which will build everything.
You can build individual directories, but you
have to install the core jars first (i.e. go to
giraph-core and do 'mvn clean install'). Then you
can build the directory of your choice.
Hope that helps,
Avery
On 5/5/13 11:11 AM, Kiru Pakkirisamy wrote:
Hi,
I am unable to compile giraph-examples because
it is not able to reach the core jar files on
the repo. Why doesn't it pick it up from the
root build dir ?
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
<http://webcloudtech.wordpress.com>
