Hi Deb, This thread is for the out-of-bound error you described. I don't think the number of iterations has any effect here. My questions were:
1) Are you using the master branch or a particular commit? 2) Do you have negative or out-of-integer-range user or product ids? Try to print out the max/min value of user/product ids. Best, Xiangrui On Sun, Apr 6, 2014 at 11:01 PM, Debasish Das <debasish.da...@gmail.com> wrote: > Hi Xiangrui, > > With 4 ALS iterations it runs fine...If I run 10 I am failing...I believe I > have to cut the lineage chain and call checkpoint....Trying to follow the > other email chain on checkpointing... > > Thanks. > Deb > > > On Sun, Apr 6, 2014 at 9:08 PM, Xiangrui Meng <men...@gmail.com> wrote: > >> Hi Deb, >> >> Are you using the master branch or a particular commit? Do you have >> negative or out-of-integer-range user or product ids? There is an >> issue with ALS' partitioning >> (https://spark-project.atlassian.net/browse/SPARK-1281), but I'm not >> sure whether that is the reason. Could you try to see whether you can >> reproduce the error on a public data set, e.g., movielens? Thanks! >> >> Best, >> Xiangrui >> >> On Sat, Apr 5, 2014 at 10:53 PM, Debasish Das <debasish.da...@gmail.com> >> wrote: >> > Hi, >> > >> > I deployed apache/spark master today and recently there were many ALS >> > related checkins and enhancements.. >> > >> > I am running ALS with explicit feedback and I remember most enhancements >> > were related to implicit feedback... >> > >> > With 25 factors my runs were successful but with 50 factors I am getting >> > array index out of bound... >> > >> > Note that I was hitting gc errors before with an older version of spark >> but >> > it seems like the sparse matrix partitioning scheme has changed >> now...data >> > caching looks much balanced now...earlier one node was becoming >> > bottleneck...Although I ran with 64g memory per node... >> > >> > There are around 3M products, 25M users... >> > >> > Anyone noticed this bug or something similar ? >> > >> > 14/04/05 23:03:15 WARN TaskSetManager: Loss was due to >> > java.lang.ArrayIndexOutOfBoundsException >> > java.lang.ArrayIndexOutOfBoundsException: 81029 >> > at >> > >> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateBlock$1$$anonfun$apply$mcVI$sp$1.apply$mcVI$sp(ALS.scala:450) >> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) >> > at >> > >> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateBlock$1.apply$mcVI$sp(ALS.scala:446) >> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) >> > at org.apache.spark.mllib.recommendation.ALS.org >> > $apache$spark$mllib$recommendation$ALS$$updateBlock(ALS.scala:445) >> > at >> > >> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:416) >> > at >> > >> org.apache.spark.mllib.recommendation.ALS$$anonfun$org$apache$spark$mllib$recommendation$ALS$$updateFeatures$2.apply(ALS.scala:415) >> > at >> > >> org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31) >> > at >> > >> org.apache.spark.rdd.MappedValuesRDD$$anonfun$compute$1.apply(MappedValuesRDD.scala:31) >> > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >> > at >> > >> org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:149) >> > at >> > >> org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:147) >> > at >> > >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >> > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >> > at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:147) >> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:229) >> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:220) >> > at >> > org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31) >> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:229) >> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:220) >> > at >> > >> org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:31) >> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:229) >> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:220) >> > at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33) >> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:229) >> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:220) >> > at >> > >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161) >> > at >> > >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102) >> > at org.apache.spark.scheduler.Task.run(Task.scala:52) >> > at >> > >> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211) >> > at >> > >> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:43) >> > at >> > >> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> > at >> > >> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:42) >> > at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > Thanks. >> > Deb >>
