Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20531#discussion_r166654955
--- Diff: python/pyspark/sql/tests.py ---
@@ -4562,14 +4585,14 @@ def test_basic(self):
self.assertPandasEqual(expected4.toPandas(), result4.toPandas())
def test_unsupported_types(self):
- from pyspark.sql.types import ArrayType, DoubleType, MapType
+ from pyspark.sql.types import DoubleType, MapType
from pyspark.sql.functions import pandas_udf, PandasUDFType
with QuietTest(self.sc):
with self.assertRaisesRegexp(NotImplementedError, 'not
supported'):
- @pandas_udf(ArrayType(DoubleType()),
PandasUDFType.GROUPED_AGG)
+ @pandas_udf(ArrayType(ArrayType(TimestampType())),
PandasUDFType.GROUPED_AGG)
--- End diff --
Seems because we don't handle the timezone issue when it's nested. There
are few todos, for example:
https://github.com/apache/spark/blob/71cfba04aeec5ae9b85a507b13996e80f8750edc/python/pyspark/sql/session.py#L465
https://github.com/apache/spark/blob/a24c03138a6935a442b983c8a4c721b26df3f9e2/python/pyspark/sql/types.py#L1726
https://github.com/apache/spark/blob/a24c03138a6935a442b983c8a4c721b26df3f9e2/python/pyspark/sql/types.py#L1745
https://github.com/apache/spark/blob/a24c03138a6935a442b983c8a4c721b26df3f9e2/python/pyspark/sql/types.py#L1771
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
