Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19884
Ok I did some local testing with these changes and pyarrow 0.8.0 with
different combinations of Python and Pandas:
**python 3.6.3, pandas 0.19.2**
ERROR: test_createDataFrame_does_not_modify_input
(pyspark.sql.tests.ArrowTests)
```
pyarrow.lib.ArrowInvalid: Casting from timestamp[ns] to timestamp[us,
tz=UTC] would lose data: 28800000000001
```
**python 3.6.3, pandas 0.21.0**
All tests pass
**python 2.7.14, pandas 0.21.1**
All tests pass
**python 2.7.14, pandas 0.19.2**
ERROR: test_createDataFrame_does_not_modify_input
(pyspark.sql.tests.ArrowTests)
```
pyarrow.lib.ArrowInvalid: Casting from timestamp[ns] to timestamp[us,
tz=UTC] would lose data: 28800000000001
```
It seems like pandas 0.19.2 has a timestamp issue, but let's see if it is
reproduced in the Jenkins env here
cc @wesm
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
