Github user icexelloss commented on the issue:

    https://github.com/apache/spark/pull/20567
  
    Sorry I am late to the party. 
https://github.com/apache/spark/pull/20567#issuecomment-364639922 Does look 
like a bug to me. However, I am a bit concerned that such magic behavior would 
be not ideal to some users. At least from python users at Two Sigma, most of 
they would prefer a "fail fast" exception rather than fall back to non-Arrow 
path as non-Arrow path, because the non-Arrow path could often take a long time 
to complete, or worse, "fail slow". Implementing this behavior and not having a 
way to turn it off could be problematic for users that transfers non trivial 
amounts of data from Spark to Pandas.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to