Bryan Cutler created SPARK-21766:
------------------------------------
Summary: DataFrame toPandas()
Key: SPARK-21766
URL: https://issues.apache.org/jira/browse/SPARK-21766
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.3.0
Reporter: Bryan Cutler
When calling {{DataFrame.toPandas()}} (without Arrow enabled), if there is a
IntegerType column that has null values the following exception is thrown:
{noformat}
ValueError: Cannot convert non-finite values (NA or inf) to integer
{noformat}
This is because the null values first get converted to float NaN during the
construction of the Pandas DataFrame in {{from_records}}, and then it is
attempted to be converted back to to an integer where it fails.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
