1) I download the imdb data from http://komarix.org/ac/ds/Blanc__Mel.txt.bz2 and use this data to test LBFGS When I run examples referencing http://spark.apache.org/docs/latest/mllib-optimization.html, an error occus. 4/07/07 08:37:27 ERROR Executor: Exception in task ID 2 java.lang.ArrayIndexOutOfBoundsException: -1 at breeze.linalg.operators.DenseVector_SparseVector_Ops$$anon$129.apply(SparseVectorOps.scala:231) at breeze.linalg.operators.DenseVector_SparseVector_Ops$$anon$129.apply(SparseVectorOps.scala:216) at breeze.linalg.operators.BinaryRegistry$class.apply(BinaryOp.scala:60) at breeze.linalg.VectorOps$$anon$178.apply(Vector.scala:391) at breeze.linalg.NumericOps$class.dot(NumericOps.scala:83) at breeze.linalg.DenseVector.dot(DenseVector.scala:47) ..................
2) I find the imdb data are zero-based-index data 0 0:1 3:1 6208:1 8936:1 8959:1 16434:1 29840:1 29843:1 30274:1 32092:1 63727:1 109302:1 114311:1 114336:1 119637:1 121867:1 143744:1 145106:1 186951:1 216401:1 228548:1 248919:1 251691:1 294713:1 302316:1 307685:1 316421:1 316556:1 317062:1 321771:1 327174:1 364381:1 384514:1 404531:1 414947:1 434235:1 434250:1 462625:1 471013:1 503923:1 511725:1 514582:1 514635:1 519251:1 524274:1 540734:1 556018:1 559036:1 559037:1 559039:1 559341:1 609032:1 644534:1 650763:1 659114:1 666864:1 669778:1 669783:1 669787:1 673083:1 3) If change code "val index = indexAndValue(0).toInt - 1" to "val index = indexAndValue(0).toInt - offset" (offset equals 0 or 1 based on user's selection), then MLUtils.loadLibSVMFile will support both zero-based-index data and one-based-index data. That also means the interface of MLUtils.loadLibSVMFile will be changed
