[
https://issues.apache.org/jira/browse/FLINK-2412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648920#comment-14648920
]
Ufuk Celebi commented on FLINK-2412:
------------------------------------
Hey Andra,
these are two seperate issues. The failure I've fixed is indeed fixed. Thanks
for your help and the traces and logs.
The other Exception occurs in the serializer:
{code}
java.lang.Exception: The data preparation for task 'CoGroup (CoGroup at
groupReduceOnNeighbors(Graph.java:1405))' , caused an error: Error obtaining
the sorted input: Thread 'SortMerger spilling thread' terminated due to an
exception: Index: 53, Size: 0
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:476)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:366)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:576)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread
'SortMerger spilling thread' terminated due to an exception: Index: 53, Size: 0
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
at
org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1096)
at
org.apache.flink.runtime.operators.CoGroupDriver.prepare(CoGroupDriver.java:98)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:471)
... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated
due to an exception: Index: 53, Size: 0
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)
Caused by: java.lang.IndexOutOfBoundsException: Index: 53, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:604)
at java.util.ArrayList.get(ArrayList.java:382)
at
com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759)
at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:127)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:30)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:127)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:30)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:531)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
{code}
I will go ahead and push my fix and open a separate issue for the new problem.
In general, I think it's easier to track problems with one issue per problem.
In your case both were IndexOutOfBoundsExceptions, but unrelated ones.
> Index Out of Bounds Exception
> -----------------------------
>
> Key: FLINK-2412
> URL: https://issues.apache.org/jira/browse/FLINK-2412
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
> Affects Versions: 0.9, 0.10
> Reporter: Andra Lungu
> Assignee: Ufuk Celebi
> Priority: Critical
> Fix For: 0.10, 0.9.1
>
>
> When running a code as simple as:
> {noformat}
> ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
> DataSet<Edge<String, NullValue>> edges = getEdgesDataSet(env);
> Graph<String, NullValue, NullValue> graph =
> Graph.fromDataSet(edges, env);
> DataSet<Tuple2<String, Long>> degrees = graph.getDegrees();
> degrees.writeAsCsv(outputPath, "\n", " ");
> env.execute();
> on the Freindster data set:
> https://snap.stanford.edu/data/com-Friendster.html; on 30 Wally nodes
>
> I get the following exception:
> java.lang.Exception: The data preparation for task 'CoGroup (CoGroup at
> inDegrees(Graph.java:701))' , caused an error: Error obtaining the sorted
> input: Thread 'SortMerger Reading Thread' terminated due to an exception:
> Fatal error at remote task manager
> 'wally028.cit.tu-berlin.de/130.149.249.38:53730'.
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:471)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
> Thread 'SortMerger Reading Thread' terminated due to an exception: Fatal
> error at remote task manager 'wally028.cit.tu-berlin.de/130.149.249.38:53730'.
> at
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
> at
> org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1145)
> at
> org.apache.flink.runtime.operators.CoGroupDriver.prepare(CoGroupDriver.java:98)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
> ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated
> due to an exception: Fatal error at remote task manager
> 'wally028.cit.tu-berlin.de/130.149.249.38:53730'.
> at
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)
> Caused by:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Fatal error at remote task manager
> 'wally028.cit.tu-berlin.de/130.149.249.38:53730'.
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.decodeMsg(PartitionRequestClientHandler.java:227)
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelRead(PartitionRequestClientHandler.java:162)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Index: 133, Size: 0
> {noformat}
> Code works fine for the twitter data set, for instance, which is bigger in
> size, but contains less vertices.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
