Burak Yavuz created SPARK-18634:
-----------------------------------
Summary: Issues with exploding Python UDFs
Key: SPARK-18634
URL: https://issues.apache.org/jira/browse/SPARK-18634
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 2.0.2, 2.1.0
Reporter: Burak Yavuz
There are some weird issues with exploding Python UDFs in SparkSQL.
There are 2 cases where based on the DataType of the exploded column, the
result can be flat out wrong, or corrupt. Seems like something bad is happening
when telling Tungsten the schema of the rows during or after applying the UDF.
Please check the attached notebook for reproduction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
