Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16537
@rdblue i think we're maybe understanding different type checks. My
understanding is in this case the error is already thrown right away. It's also
not that the user needs to pass a callable here, were checking that the UDF is
called with column or strings as arguments.
I agree the current error message is a bit obtuse to new users, but I also
don't want us to go adding type checking to every individual function wrapper
to Py4J in another PR - that just isn't scalable given the level of committer
bandwidth available in Python. I'm think we should prioritize issues that users
have run into or figure out a more scalable way to solve this (either bigger
batches of cleanups, improved explanations of Py4J errors possible causes, or
type hints - although as zero323 points out that's a larger discussion). (And
for this type check I didn't see any posts related to it).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
