Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/15821
Thanks @cloud-fan. I commented above on the reason for the type
differences, but basically without arrow `IntegerType` and `FloatType` were
getting up-converted to `int64` and `float64`. Even though this shouldn't
change any data maybe it would be good to document this change somewhere?
@leifwalsh I also added a check for `concat_tables()` in case all records
are filtered out and tables are None. It will then produce the same
pandas.DataFrame as without using Arrow, which has columns defined but is empty.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
