Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20567
> The binary type bug sounds like a blocker, can we just fix it surgically
by checking the supported data types before going to the arrow optimization
path? For now we can stick with that the current behavior is, i.e. throw
exception.
That's basically
(https://github.com/apache/spark/pull/20567#issuecomment-365064243):
```python
if # 'spark.sql.execution.arrow.enabled' true?
require_minimum_pyarrow_version()
try:
to_arrow_schema(self.schema)
# return the one with Arrow
except Exception as e:
raise Exception("'spark.sql.execution.arrow.enabled' blah blah ...")
else:
# return the one without Arrow
```
because `to_arrow_schema(self.schema)` checks the supported types like
other Pandas/Arrow functionalities.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
