Hi,
I have faced a similar issue when trying to run a map function with
predict. In my case I had some non-serializable fields in my calling class.
After making those fields transient, the error went away.
On Wed, Aug 13, 2014 at 6:39 PM, lancezhange <lancezha...@gmail.com> wrote:
> let's say you have a model which is of class
> "org.apache.spark.mllib.classification.LogisticRegressionModel"
> you can save model to disk as following:
>
> /import java.io.FileOutputStream
> import java.io.ObjectOutputStream
> val fos = new FileOutputStream("e:/model.obj")
> val oos = new ObjectOutputStream(fos)
> oos.writeObject(model)
> oos.close/
>
> and load it in:
> /import java.io.FileInputStream
> import java.io.ObjectInputStream
> val fos = new FileInputStream("e:/model.obj")
> val oos = new ObjectInputStream(fos)
> val newModel =
>
> oos.readObject().asInstanceOf[org.apache.spark.mllib.classification.LogisticRegressionModel]/
>
> you can check that '/newModel.weights/' gives you the weights, implying
> that
> newModel is loaded successfully.
>
> There remains, however, another problem, which confuses me badly: when i
> use
> the loaded newModel to predict on LabeledPoints, there is always a "Task
> not
> serializable" exception! Detailed logs:
> INFO DAGScheduler: Failed to run count at <console>:49
> org.apache.spark.SparkException: Job aborted due to stage failure: Task not
> serializable: java.io.NotSeri
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndInd
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$submitMissing
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$submitStage(D
> at
>
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:697)
> at
>
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGSch
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219) in 2646 ms on
> localhost (progress: 345/345)
> at
>
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)ed in
> 528.389 s
> at
>
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)ed,
> from pool
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> Any help here?
> PS. any one knows the *constructor function* of the model assuming you
> have
> weights and intercept?
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12030.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
