Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20567 The binary type bug sounds like a blocker, can we just fix it surgically by checking the supported data types before going to the arrow optimization path? For now we can stick with that the current behavior is, i.e. throw exception. The inconsistent behavior between `toPandas` and `createDataFrame` is confusing but may not be a blocker. We can fix it in Spark 2.4 and add a note in the migration guide.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org