Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20908#discussion_r177294401
--- Diff: python/pyspark/sql/tests.py ---
@@ -3966,6 +3967,15 @@ def random_udf(v):
random_udf = random_udf.asNondeterministic()
return random_udf
+ def test_pandas_udf_tokenize(self):
+ from pyspark.sql.functions import pandas_udf
+ tokenize = pandas_udf(lambda s: s.apply(lambda str: str.split('
')),
--- End diff --
hm. I thought this PR targets to clarify array type wuth primitive types.
can we improve the test case here
-https://github.com/holdenk/spark/blob/342d2228a5c68fd2c07bd8c1b518da6135ce1bf6/python/pyspark/sql/tests.py#L3998,
and remove this test case?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
