Hi everyone. I'm running the FlinkML ALS matrix factorization and I bumped into the following exception:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.Client.runBlocking(Client.java:381) at org.apache.flink.client.program.Client.runBlocking(Client.java:355) at org.apache.flink.client.program.Client.runBlocking(Client.java:315) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:60) at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:652) at org.apache.flink.ml.common.FlinkMLTools$.persist(FlinkMLTools.scala:94) at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:507) at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:433) at org.apache.flink.ml.pipeline.Estimator$class.fit(Estimator.scala:55) at org.apache.flink.ml.recommendation.ALS.fit(ALS.scala:122) at dima.tu.berlin.benchmark.flink.als.RUN$.main(RUN.scala:78) at dima.tu.berlin.benchmark.flink.als.RUN.main(RUN.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) at org.apache.flink.client.program.Client.runBlocking(Client.java:248) at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1192) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243) Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:717) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:663) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:663) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.RuntimeException: Initializing the input processing failed: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: null at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:325) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: null at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619) at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1079) at org.apache.flink.runtime.operators.BatchTask.initLocalStrategies(BatchTask.java:819) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:321) ... 2 more Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800) Caused by: org.apache.flink.runtime.io.network.partition.ProducerFailedException at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.getNextLookAhead(LocalInputChannel.java:270) at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.onNotification(LocalInputChannel.java:238) at org.apache.flink.runtime.io.network.partition.PipelinedSubpartition.release(PipelinedSubpartition.java:158) at org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:320) at org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartitionsProducedBy(ResultPartitionManager.java:95) at org.apache.flink.runtime.io.network.NetworkEnvironment.unregisterTask(NetworkEnvironment.java:370) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:657) at java.lang.Thread.run(Thread.java:745) I'm running with* flink-1.0.3*. I really can't figure out the reason behind that. My code simply calls the library as follows: val als = ALS() .setIterations(numIterations) .setNumFactors(rank) .setBlocks(degreeOfParallelism) .setSeed(42) .setTemporaryPath(tempPath) als.fit(ratings, parameters) val (users, items) = als.factorsOption match { case Some(_) => als.factorsOption.get case _ => throw new RuntimeException } users.writeAsText(outputPath, WriteMode.OVERWRITE) items.writeAsText(outputPath, WriteMode.OVERWRITE) env.execute("ALS matrix factorization") where - ratings as the input dataset contains (uid, iid, rate) rows about 8e6 users, 1e6 items and 700 rating per user average. - numIterations 10 - rank 50 - degreeOfParallelism 240 *The error seems to be related to the final .persists() call.*at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:507) I'm running with a 15 nodes cluster - 16cpus per node - with the following valuable properties: jobmanager.heap.mb = 2048 taskmanager.memory.fraction = 0.5 taskmanager.heap.mb = 28672 taskmanager.network.bufferSizeInBytes = 32768 taskmanager.network.numberOfBuffers = 98304 akka.ask.timeout = 300s Any help will be appreciated. Thank you. -- *Andrea Spina* N.Tessera: *74598* MAT: *89369* *Ingegneria Informatica* *[LM] *(D.M. 270)