zable?
>>>
>>>
>>>
>>> Just thinking out loud
>>>
>>>
>>>
>>> Simon Scott
>>>
>>>
>>>
>>> Research Developer @ viavisolutions.com
>>>
>>>
>>>
>>> *From:*
Ints are
>> serializable?
>>
>>
>>
>> Just thinking out loud
>>
>>
>>
>> Simon Scott
>>
>>
>>
>> Research Developer @ viavisolutions.com
>>
>>
>>
>> *From:* Hao Ren [mailto:inv...@gmail.com]
>> *Sent:
@ viavisolutions.com
>
>
>
> *From:* Hao Ren [mailto:inv...@gmail.com]
> *Sent:* 08 August 2016 09:03
> *To:* Muthu Jayakumar
> *Cc:* user ; dev
> *Subject:* Re: [SPARK-2.0][SQL] UDF containing non-serializable object
> does not work as expected
>
>
>
> Yes, it
Yes, it is.
You can define a udf like that.
Basically, it's a udf Int => Int which is a closure contains a non
serializable object.
The latter should cause Task not serializable exception.
Hao
On Mon, Aug 8, 2016 at 5:08 AM, Muthu Jayakumar wrote:
> Hello Hao Ren,
>
> Doesn't the code...
>
> va
Hello Hao Ren,
Doesn't the code...
val add = udf {
(a: Int) => a + notSer.value
}
Mean UDF function that Int => Int ?
Thanks,
Muthu
On Sun, Aug 7, 2016 at 2:31 PM, Hao Ren wrote:
> I am playing with spark 2.0
> What I tried to test is:
>
> Create a UDF in which there is a non serial
I am playing with spark 2.0
What I tried to test is:
Create a UDF in which there is a non serializable object.
What I expected is when this UDF is called during materializing the
dataFrame where the UDF is used in "select", an task non serializable
exception should be thrown.
It depends also which
