hi,
i want to run a multiclass classification with 390 classes on120k label
points(tf-idf vectors). but i get the following exception. If i reduce the
number of classes to ~20 everythings work fine. How can i fix this?
i use the LogisticRegressionWithLBFGS class for my classification on a 8
Node Cluster with
total-executor-cores = 30
executor-memory = 20g
My Exception:
15/07/02 15:55:00 INFO DAGScheduler: Job 11 finished: count at
LBFGS.scala:170, took 0,521823 s
15/07/02 15:55:02 INFO MemoryStore: ensureFreeSpace(-1069858488) called with
curMem=308280107, maxMem=11113699737
15/07/02 15:55:02 INFO MemoryStore: Block broadcast_22 stored as values in
memory (estimated size -1069858488.0 B, free 11.1 GB)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed:
sizeInBytes was negative: -1069858488
at scala.Predef$.require(Predef.scala:233)
at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:812)
at
org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:635)
at
org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:993)
at
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:99)
at
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85)
at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1289)
at
org.apache.spark.mllib.optimization.LBFGS$CostFun.calculate(LBFGS.scala:215)
at
org.apache.spark.mllib.optimization.LBFGS$CostFun.calculate(LBFGS.scala:204)
at
breeze.optimize.CachedDiffFunction.calculate(CachedDiffFunction.scala:23)
at
breeze.optimize.FirstOrderMinimizer.calculateObjective(FirstOrderMinimizer.scala:108)
at
breeze.optimize.FirstOrderMinimizer.initialState(FirstOrderMinimizer.scala:101)
at
breeze.optimize.FirstOrderMinimizer.iterations(FirstOrderMinimizer.scala:146)
at org.apache.spark.mllib.optimization.LBFGS$.runLBFGS(LBFGS.scala:178)
at org.apache.spark.mllib.optimization.LBFGS.optimize(LBFGS.scala:117)
at
org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:282)
at
org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:205)
at
com.test.spark.SVMSimpleAppEC2$.createNaiveBayesModel(SVMSimpleAppEC2.scala:150)
at com.test.spark.SVMSimpleAppEC2$.main(SVMSimpleAppEC2.scala:48)
at com.test.spark.SVMSimpleAppEC2.main(SVMSimpleAppEC2.scala)
... 6 more
15/07/02 15:55:02 INFO SparkContext: Invoking stop() from shutdown hook
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-MLLib-Bug-Multiclass-Classification-requirement-failed-sizeInBytes-was-negative-tp23610.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
