Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/20567
Sorry I am late to the party.
https://github.com/apache/spark/pull/20567#issuecomment-364639922 Does look
like a bug to me. However, I am a bit concerned that such magic behavior would
be not ideal to some users. At least from python users at Two Sigma, most of
they would prefer a "fail fast" exception rather than fall back to non-Arrow
path as non-Arrow path, because the non-Arrow path could often take a long time
to complete, or worse, "fail slow". Implementing this behavior and not having a
way to turn it off could be problematic for users that transfers non trivial
amounts of data from Spark to Pandas.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
