Could you put the implicit def in an object? That should work, as objects are never serialized.
On Wed, Apr 29, 2015 at 6:28 PM, guoqing0...@yahoo.com.hk < guoqing0...@yahoo.com.hk> wrote: > Thank you for your pointers , it`s very helpful to me , in this scenario > how can i use the implicit def in the enclosing class ? > > > > *From:* Tathagata Das <t...@databricks.com> > *Date:* 2015-04-30 07:00 > *To:* guoqing0...@yahoo.com.hk > *CC:* user <user@spark.apache.org> > *Subject:* Re: implicit function in SparkStreaming > I believe that the implicit def is pulling in the enclosing class (in > which the def is defined) in the closure which is not serializable. > > > On Wed, Apr 29, 2015 at 4:20 AM, guoqing0...@yahoo.com.hk < > guoqing0...@yahoo.com.hk> wrote: > >> Hi guys, >> I`m puzzled why i cant use the implicit function in spark streaming to >> cause Task not serializable . >> >> code snippet: >> >> implicit final def str2KeyValue(s:String): (String,String) = { >> val message = s.split("\\|") >> if(message.length >= 2) >> (message(0),message(1)) >> else if(message.length == 1) { >> (message(0), "") >> } >> else >> ("","") >> } >> >> def filter(stream:DStream[String]) :DStream[String] = { >> stream.filter(s => { >> (s._1=="Action" && s._2=="TRUE") >> }) >> >> >> Could you please give me some pointers ? Thank you . >> >> ------------------------------ >> guoqing0...@yahoo.com.hk >> > >