Appreciate for your help , it works . i`m curious why the enclosing class
cannot serialized , is it need to extends java.io.Serializable ? if object
never serialized how it works in the task .whether there`s any association with
the spark.closure.serializer .
guoqing0...@yahoo.com.hk
From: Tathagata Das
Date: 2015-04-30 09:30
To: guoqing0...@yahoo.com.hk
CC: user
Subject: Re: Re: implicit function in SparkStreaming
Could you put the implicit def in an object? That should work, as objects are
never serialized.
On Wed, Apr 29, 2015 at 6:28 PM, guoqing0...@yahoo.com.hk
<guoqing0...@yahoo.com.hk> wrote:
Thank you for your pointers , it`s very helpful to me , in this scenario how
can i use the implicit def in the enclosing class ?
From: Tathagata Das
Date: 2015-04-30 07:00
To: guoqing0...@yahoo.com.hk
CC: user
Subject: Re: implicit function in SparkStreaming
I believe that the implicit def is pulling in the enclosing class (in which the
def is defined) in the closure which is not serializable.
On Wed, Apr 29, 2015 at 4:20 AM, guoqing0...@yahoo.com.hk
<guoqing0...@yahoo.com.hk> wrote:
Hi guys,
I`m puzzled why i cant use the implicit function in spark streaming to cause
Task not serializable .
code snippet:
implicit final def str2KeyValue(s:String): (String,String) = {
val message = s.split("\\|")
if(message.length >= 2)
(message(0),message(1))
else if(message.length == 1) {
(message(0), "")
}
else
("","")
}
def filter(stream:DStream[String]) :DStream[String] = {
stream.filter(s => {
(s._1=="Action" && s._2=="TRUE")
})
Could you please give me some pointers ? Thank you .
guoqing0...@yahoo.com.hk
