Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
> he only other idea I have is to provide an option to raise an error if
the type needs to be cast.
Actually sounds good to me.
I think the problem is we are not quite clear about when the type is
mismatched in UDFs (see also https://github.com/apache/spark/pull/20163 for a
reminder). IIRC, we rather roughly agreed upon documenting it (and allowing
exact type match (?)).
@viirya and @BryanCutler, how about we document that return types should be
matched (we can leave a chart or map referring (`types.to_arrow_type`)?
One additional improvement might be .. we describe that type casting
behaviour is .. say .. not guaranteed but I am not sure how we can nicely
document this. Probably only mentioning the type mapping is fine ..?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
