srowen commented on issue #22807: [SPARK-25811][PySpark] Raise a proper error when unsafe cast is detected by PyArrow URL: https://github.com/apache/spark/pull/22807#issuecomment-454036714 So, in these situations, we agree some error should occur. The current one is an overflow error -- right? or does it silently overfloat? Non-arrow UDFs seem to do the latter. Based on that, I'm not clear the Arrow behavior should be different, especially if it already matches that behavior, and is what Arrow users expect. No, this isn't a case for a flag. That is a failure to decide, punted to the user.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
