Laurens created SPARK-34693: ------------------------------- Summary: Support null in conversions to and from Arrow Key: SPARK-34693 URL: https://issues.apache.org/jira/browse/SPARK-34693 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.1.1 Reporter: Laurens
Looks like a regression from or related to SPARK-33489. I got the following running spark 3.1.1. with koalas 1.7.0 {code:java} TypeError Traceback (most recent call last) /var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/udf.py in returnType(self) 100 try: --> 101 to_arrow_type(self._returnType_placeholder) 102 except TypeError: /var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/pandas/types.py in to_arrow_type(dt) 75 else: ---> 76 raise TypeError("Unsupported type in conversion to Arrow: " + str(dt)) 77 return arrow_type TypeError: Unsupported type in conversion to Arrow: NullType {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org