Github user rednaxelafx commented on the issue:
https://github.com/apache/spark/pull/20163
Given the above discussion, do we have consensus on all of the following:
- Update the documentation for PySpark UDFs to warn about the behavior of
mismatched declared `returnType` vs actual runtime return values
- Make Python UDFs that declared `returnType` as `StringType` recognize
`java.util.Calendar` and convert the value into a `null` (as in the example
from @HyukjinKwon ), essentially marking it as unconvertible.
I believe we all agree on the first point. The second point above is in
line with @icexelloss 's opinion, which I tend to agree in terms of API
semantic consistency. It might not be as user-friendly as Option 2 from
@HyukjinKwon , but it's less magic and more consistent. I tend to find more
consistency leads to less surprises.
If we have consensus then I'll update the JIRA ticket and this PR to
reflect that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
