Re: Re: implicit function in SparkStreaming

[Lin Hao Xu]

For you question, I think the discussion in this link can help.

http://apache-spark-user-list.1001560.n3.nabble.com/Error-related-to-serialisation-in-spark-streaming-td6801.html

Best regards,

Lin Hao XU
IBM Research China
Email: xulin...@cn.ibm.com
My Flickr: http://www.flickr.com/photos/xulinhao/sets



From:   "guoqing0...@yahoo.com.hk" <guoqing0...@yahoo.com.hk>
To:     "Tathagata Das" <t...@databricks.com>
Cc:     user <user@spark.apache.org>
Date:   2015/04/30 11:17
Subject:        Re: Re: implicit function in SparkStreaming



Appreciate for your help , it works . i`m curious why the enclosing class
cannot serialized , is it need to extends  java.io.Serializable ? if object
never serialized how it works in the task .whether there`s any association
with the spark.closure.serializer .


 guoqing0...@yahoo.com.hk

 From: Tathagata Das
 Date: 2015-04-30 09:30
 To: guoqing0...@yahoo.com.hk
 CC: user
 Subject: Re: Re: implicit function in SparkStreaming
 Could you put the implicit def in an object? That should work, as objects
 are never serialized.


 On Wed, Apr 29, 2015 at 6:28 PM, guoqing0...@yahoo.com.hk <
 guoqing0...@yahoo.com.hk> wrote:
   Thank you for your pointers , it`s very helpful to me , in this scenario
   how can i use the implicit def in the enclosing class ?



    From: Tathagata Das
    Date: 2015-04-30 07:00
    To: guoqing0...@yahoo.com.hk
    CC: user
    Subject: Re: implicit function in SparkStreaming
    I believe that the implicit def is pulling in the enclosing class (in
    which the def is defined) in the closure which is not serializable.


    On Wed, Apr 29, 2015 at 4:20 AM, guoqing0...@yahoo.com.hk <
    guoqing0...@yahoo.com.hk> wrote:
     Hi guys,
     I`m puzzled why i cant use the implicit function in spark streaming to
     cause Task not serializable .

     code snippet:
     implicit final def str2KeyValue(s:String): (String,String) = {
       val message = s.split("\\|")
       if(message.length >= 2)
         (message(0),message(1))
       else if(message.length == 1) {
         (message(0), "")
       }
       else
         ("","")
     }

     def filter(stream:DStream[String]) :DStream[String] = {
       stream.filter(s => {
         (s._1=="Action" && s._2=="TRUE")
       })

     Could you please give me some pointers ? Thank you .


      guoqing0...@yahoo.com.hk