Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21427
Pandas UDF is already in 2 releases(2.3.0 and 2.3.1), we can't just change
the behavior. We think the old behavior doesn't make sense and users should
change their code, but users may not think in this way.
Educating users takes times, I think providing a config can make it smooth.
Note that, this case is special: we turn runnable code into failure, and
the old behavior is explainable(by-position match). I don't think there are a
lot of behavior changes like this, so having version-specific configs seems an
overkill.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
