I think in the following case
class Foo { def foo() = Array(1.0) }
val t = new Foo
val m = t.foo
val r1 = sc.parallelize(List(1, 2, 3))
val r2 = r1.map(_ + m(0))
r2.toArray
Spark should not serialize "t". But looks it will.
Best Regards,
Shixiong Zhu
2014-08-14 23:22 GMT+08:00 lancezhange <lancezha...@gmail.com>:
> Following codes works, too
>
> class Foo1 extends Serializable { def foo() = Array(1.0) }
> val t1 = new Foo1
> val m1 = t1.foo
> val r11 = sc.parallelize(List(1, 2, 3))
> val r22 = r11.map(_ + m1(0))
> r22.toArray
>
>
>
>
>
> On Thu, Aug 14, 2014 at 10:55 PM, Shixiong Zhu [via Apache Spark User
> List] <[hidden email] <http://user/SendEmail.jtp?type=node&node=12114&i=0>
> > wrote:
>
>> I think I can reproduce this error.
>>
>> The following code cannot work and report "Foo" cannot be
>> serialized. (log in gist
>> https://gist.github.com/zsxwing/4f9f17201d4378fe3e16):
>>
>> class Foo { def foo() = Array(1.0) }
>> val t = new Foo
>> val m = t.foo
>> val r1 = sc.parallelize(List(1, 2, 3))
>> val r2 = r1.map(_ + m(0))
>> r2.toArray
>>
>> But the following code can work (log in gist
>> https://gist.github.com/zsxwing/802cade0facb36a37656):
>>
>> class Foo { def foo() = Array(1.0) }
>> var m: Array[Double] = null
>> {
>> val t = new Foo
>> m = t.foo
>> }
>> val r1 = sc.parallelize(List(1, 2, 3))
>> val r2 = r1.map(_ + m(0))
>> r2.toArray
>>
>>
>> Best Regards,
>> Shixiong Zhu
>>
>>
>> 2014-08-14 22:11 GMT+08:00 Christopher Nguyen <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=12112&i=0>>:
>>
>>> Hi Hoai-Thu, the issue of private default constructor is unlikely the
>>> cause here, since Lance was already able to load/deserialize the model
>>> object.
>>>
>>> And on that side topic, I wish all serdes libraries would just use
>>> constructor.setAccessible(true) by default :-) Most of the time that
>>> privacy is not about serdes reflection restrictions.
>>>
>>> Sent while mobile. Pls excuse typos etc.
>>> On Aug 14, 2014 1:58 AM, "Hoai-Thu Vuong" <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=12112&i=1>> wrote:
>>>
>>>> A man in this community give me a video:
>>>> https://www.youtube.com/watch?v=sPhyePwo7FA. I've got a same question
>>>> in this community and other guys helped me to solve this problem. I'm
>>>> trying to load MatrixFactorizationModel from object file, but compiler said
>>>> that, I can not create object because the constructor is private. To solve
>>>> this, I put my new object to same package as MatrixFactorizationModel.
>>>> Luckly it works.
>>>>
>>>>
>>>> On Wed, Aug 13, 2014 at 9:20 PM, Christopher Nguyen <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=2>> wrote:
>>>>
>>>>> Lance, some debugging ideas: you might try model.predict(RDD[Vector])
>>>>> to isolate the cause to serialization of the loaded model. And also try to
>>>>> serialize the deserialized (loaded) model "manually" to see if that throws
>>>>> any visible exceptions.
>>>>>
>>>>> Sent while mobile. Pls excuse typos etc.
>>>>> On Aug 13, 2014 7:03 AM, "lancezhange" <[hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=3>> wrote:
>>>>>
>>>>>> my prediction codes are simple enough as follows:
>>>>>>
>>>>>> *val labelsAndPredsOnGoodData = goodDataPoints.map { point =>
>>>>>> val prediction = model.predict(point.features)
>>>>>> (point.label, prediction)
>>>>>> }*
>>>>>>
>>>>>> when model is the loaded one, above code just can't work. Can you
>>>>>> catch the
>>>>>> error?
>>>>>> Thanks.
>>>>>>
>>>>>> PS. i use spark-shell under standalone mode, version 1.0.0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12035.html
>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>> Nabble.com.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=4>
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=5>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Thu.
>>>>
>>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12112.html
>> To unsubscribe from How to save mllib model to hdfs and reload it, click
>> here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
> --
> -- 张喜升
>
> ------------------------------
> View this message in context: Re: How to save mllib model to hdfs and
> reload it
> <http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12114.html>
>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
