BryanCutler commented on issue #22807: [SPARK-25811][PySpark] Raise a proper error when unsafe cast is detected by PyArrow URL: https://github.com/apache/spark/pull/22807#issuecomment-454117538 @srowen we discussed adding this flag a while ago when pyarrow 0.11.1 made this option which seemed to change default behavior in pyspark. Our intention was to preserve default behavior in pyspark and provide the user a config to change it if needed. It was only recently we learned there was a behavior change either way the config is set. It does seem like there is a bug in Arrow that is affecting this too, https://issues.apache.org/jira/browse/ARROW-4258, although we still should make a patch for v0.11.1
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
