Is there an approximate date when this new DiskBackedPartitionStore will
become available? I'd be very interested to test it.
--sebastian
On 02/10/2014 07:33 PM, Claudio Martella wrote:
Hi Sebastian,
this is indeed a bug. Apparently some updates to the PartitionStore API
introduced a regression. Armando has worked on a re-design of the
DiskBackedPartitionStore, which we will be uploaded for a review as soon as
we finish testing it.
Stay tuned.
On Mon, Feb 10, 2014 at 6:42 PM, Sebastian Schelter <[email protected]> wrote:
Hi,
I'm trying to process a huge graph on our cluster and therefore enabled
out-of-core. Unfortunately, I run into a NullPointerException during the
input phase.
What can I do to fix this? I'm using the following settings:
giraph.oneToAllMsgSending=true
giraph.isStaticGraph=true
giraph.numComputeThreads=15
giraph.numInputThreads=15
giraph.numOutputThreads=15
giraph.maxNumberOfSupersteps=30
giraph.useOutOfCoreGraph=true
Here's the log for the exception:
java.lang.IllegalStateException: run: Caught an unrecoverable exception
waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6e172129
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
occurred while waiting for org.apache.giraph.utils.ProgressableUtils$
FutureWaitable@6e172129
at org.apache.giraph.utils.ProgressableUtils.waitFor(
ProgressableUtils.java:181)
at org.apache.giraph.utils.ProgressableUtils.waitForever(
ProgressableUtils.java:139)
at org.apache.giraph.utils.ProgressableUtils.waitForever(
ProgressableUtils.java:124)
at org.apache.giraph.utils.ProgressableUtils.getFutureResult(
ProgressableUtils.java:87)
at org.apache.giraph.utils.ProgressableUtils.
getResultsWithNCallables(ProgressableUtils.java:221)
at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
BspServiceWorker.java:281)
at org.apache.giraph.worker.BspServiceWorker.loadVertices(
BspServiceWorker.java:325)
at org.apache.giraph.worker.BspServiceWorker.setup(
BspServiceWorker.java:506)
at org.apache.giraph.graph.GraphTaskManager.execute(
GraphTaskManager.java:244)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException:
getOrCreatePartition: cannot retrieve partition 14
at java.util.concurrent.FutureTask$Sync.innerGet(
FutureTask.java:262)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at org.apache.giraph.utils.ProgressableUtils$
FutureWaitable.waitFor(ProgressableUtils.java:300)
at org.apache.giraph.utils.ProgressableUtils.waitFor(
ProgressableUtils.java:173)
... 16 more
Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot
retrieve partition 14
at org.apache.giraph.partition.DiskBackedPartitionStore.
getOrCreatePartition(DiskBackedPartitionStore.java:243)
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
doRequest(SendWorkerVerticesRequest.java:110)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
sor.doRequest(NettyWorkerClientRequestProcessor.java:482)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
sor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276)
at org.apache.giraph.worker.VertexInputSplitsCallable.
readInputSplit(VertexInputSplitsCallable.java:172)
at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(
InputSplitsCallable.java:267)
at org.apache.giraph.worker.InputSplitsCallable.call(
InputSplitsCallable.java:211)
at org.apache.giraph.worker.InputSplitsCallable.call(
InputSplitsCallable.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(
LogStacktraceCallable.java:51)
at java.util.concurrent.FutureTask$Sync.innerRun(
FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(
FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.giraph.partition.DiskBackedPartitionStore.
getOrCreatePartition(DiskBackedPartitionStore.java:228)
... 13 more
Caused by: java.lang.NullPointerException
at org.apache.giraph.partition.DiskBackedPartitionStore$
GetPartition.call(DiskBackedPartitionStore.java:692)
at org.apache.giraph.partition.DiskBackedPartitionStore$
GetPartition.call(DiskBackedPartitionStore.java:658)
at java.util.concurrent.FutureTask$Sync.innerRun(
FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.apache.giraph.partition.DiskBackedPartitionStore$
DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
at java.util.concurrent.AbstractExecutorService.submit(
AbstractExecutorService.java:132)
... 14 more
