Re: Problems using UDFs with PySpark on 0.6.1 / Spark 2.0

[Michael Borst]

Thanks for the swift reply! Having the patch at least means it's not a
blocker anymore.

On Mon, Sep 19, 2016 at 9:18 PM, moon soo Lee  wrote:

> Hi,
>
> Thanks for reporting the problem.
> The issue is being tracked by https://issues.apache.org/
> jira/browse/ZEPPELIN-1411 and patch is available at
> https://github.com/apache/zeppelin/pull/1404. Hope we can merge this
> patch to master soon.
>
> Thanks!
> moon
>
> On Mon, Sep 19, 2016 at 2:58 AM Michael Borst <
> michael.bo...@innovation.hrs.com> wrote:
>
>> Hi all,
>>
>> I am experiencing problems with defining UDFs on Zeppelin 0.6.1 backed by
>> spark 2.0.
>>
>> Code:
>>
>> %pyspark
>> from pyspark.sql.functions import udf
>>
>> wrapped_udf = udf(lambda x: x)
>>
>> The call to udf fails with the following error message:
>>
>> Traceback (most recent call last):
>> File 
>> "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
>> line 266, in 
>> raise Exception(traceback.format_exc())
>> Exception: Traceback (most recent call last):
>> File 
>> "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
>> line 264, in 
>> exec(code)
>> File "", line 2, in 
>> File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
>> line 1789, in udf
>> return UserDefinedFunction(f, returnType)
>> File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
>> line 1751, in __init__
>> self._judf = self._create_judf(name)
>> File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
>> line 1758, in _create_judf
>> jdt = ctx._ssql_ctx.parseDataType(self.returnType.json())
>> AttributeError: 'JavaMember' object has no attribute 'parseDataType'
>>
>> The same code executed from a regular PySpark shell works fine. Anyone
>> having the same issues or able to provide help?
>>
>> Regards
>>
>> --
>>
>> *Michael Borst*
>> Software Engineer
>>
>> HRS Innovation Hub
>> Web  • LinkedIn
>>  • Facebook
>> 
>>
>> We're hiring! See openings 
>> Terms apply to this email: http://j.mp/email-tac
>>
>


-- 

*Michael Borst*
Software Engineer

HRS Innovation Hub
Web  • LinkedIn
 • Facebook


We're hiring! See openings 
Terms apply to this email: http://j.mp/email-tac

Re: Problems using UDFs with PySpark on 0.6.1 / Spark 2.0

[moon soo Lee]

Hi,

Thanks for reporting the problem.
The issue is being tracked by
https://issues.apache.org/jira/browse/ZEPPELIN-1411 and patch is available
at https://github.com/apache/zeppelin/pull/1404. Hope we can merge this
patch to master soon.

Thanks!
moon

On Mon, Sep 19, 2016 at 2:58 AM Michael Borst <
michael.bo...@innovation.hrs.com> wrote:

> Hi all,
>
> I am experiencing problems with defining UDFs on Zeppelin 0.6.1 backed by
> spark 2.0.
>
> Code:
>
> %pyspark
> from pyspark.sql.functions import udf
>
> wrapped_udf = udf(lambda x: x)
>
> The call to udf fails with the following error message:
>
> Traceback (most recent call last):
> File
> "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
> line 266, in 
> raise Exception(traceback.format_exc())
> Exception: Traceback (most recent call last):
> File
> "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
> line 264, in 
> exec(code)
> File "", line 2, in 
> File
> "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line
> 1789, in udf
> return UserDefinedFunction(f, returnType)
> File
> "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line
> 1751, in __init__
> self._judf = self._create_judf(name)
> File
> "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line
> 1758, in _create_judf
> jdt = ctx._ssql_ctx.parseDataType(self.returnType.json())
> AttributeError: 'JavaMember' object has no attribute 'parseDataType'
>
> The same code executed from a regular PySpark shell works fine. Anyone
> having the same issues or able to provide help?
>
> Regards
>
> --
>
> *Michael Borst*
> Software Engineer
>
> HRS Innovation Hub
> Web  • LinkedIn
>  • Facebook
> 
>
> We're hiring! See openings 
> Terms apply to this email: http://j.mp/email-tac
>

Problems using UDFs with PySpark on 0.6.1 / Spark 2.0

[Michael Borst]

Hi all,

I am experiencing problems with defining UDFs on Zeppelin 0.6.1 backed by
spark 2.0.

Code:

%pyspark
from pyspark.sql.functions import udf

wrapped_udf = udf(lambda x: x)

The call to udf fails with the following error message:

Traceback (most recent call last):
File 
"/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
line 266, in 
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
File 
"/var/folders/mr/797qcfdd0wd0vz51n4l0xmxhgn/T/zeppelin_pyspark-6894515524820639358.py",
line 264, in 
exec(code)
File "", line 2, in 
File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
line 1789, in udf
return UserDefinedFunction(f, returnType)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
line 1751, in __init__
self._judf = self._create_judf(name)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py",
line 1758, in _create_judf
jdt = ctx._ssql_ctx.parseDataType(self.returnType.json())
AttributeError: 'JavaMember' object has no attribute 'parseDataType'

The same code executed from a regular PySpark shell works fine. Anyone
having the same issues or able to provide help?

Regards

-- 

*Michael Borst*
Software Engineer

HRS Innovation Hub
Web  • LinkedIn
 • Facebook


We're hiring! See openings 
Terms apply to this email: http://j.mp/email-tac