Hi all, I am experiencing problems with defining UDFs on Zeppelin 0.6.1 backed by spark 2.0.
Code: %pyspark from pyspark.sql.functions import udf wrapped_udf = udf(lambda x: x) The call to udf fails with the following error message: Traceback (most recent call last): File "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxh0000gn/T/zeppelin_pyspark-6894515524820639358.py", line 266, in <module> raise Exception(traceback.format_exc()) Exception: Traceback (most recent call last): File "/var/folders/mr/797qcfdd0wd0vz51n4l0xmxh0000gn/T/zeppelin_pyspark-6894515524820639358.py", line 264, in <module> exec(code) File "<stdin>", line 2, in <module> File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line 1789, in udf return UserDefinedFunction(f, returnType) File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line 1751, in __init__ self._judf = self._create_judf(name) File "/usr/local/opt/apache-spark/libexec/python/pyspark/sql/functions.py", line 1758, in _create_judf jdt = ctx._ssql_ctx.parseDataType(self.returnType.json()) AttributeError: 'JavaMember' object has no attribute 'parseDataType' The same code executed from a regular PySpark shell works fine. Anyone having the same issues or able to provide help? Regards -- *Michael Borst* Software Engineer HRS Innovation Hub Web <http://innovation.hrs.com/> • LinkedIn <https://www.linkedin.com/company/hrs-innovation-hub> • Facebook <https://www.facebook.com/hrsinnovation> We're hiring! See openings <https://hrsinnovationhub.recruiterbox.com/> Terms apply to this email: http://j.mp/email-tac
