[
https://issues.apache.org/jira/browse/ARROW-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429349#comment-17429349
]
Joris Van den Bossche commented on ARROW-14004:
-----------------------------------------------
[~miwelc] late answer, but yes if you would construct the exact metadata as you
currently get from a pandas (with nullable dtypes) -> arrow conversion, that
should be a relatively robust solution. The exact metadata might change in the
future, but even then we have to provide compatibility with the current format
(since there are eg parquet files written that have the current metadata
format).
I opened a PR that start to document this better. Feedback certainly welcome!
> [Python] to_pandas() converts to float instead of using pandas nullable types
> -----------------------------------------------------------------------------
>
> Key: ARROW-14004
> URL: https://issues.apache.org/jira/browse/ARROW-14004
> Project: Apache Arrow
> Issue Type: Bug
> Components: Documentation, Python
> Reporter: Miguel Cantón Cortés
> Assignee: Joris Van den Bossche
> Priority: Major
> Labels: pandas
> Fix For: 7.0.0
>
> Attachments: image.png
>
>
> We've noticed that when converting an Arrow Table to pandas using
> `.to_pandas()` integer columns with null values get converted to float
> instead of using pandas nullable types.
> If the column was created with pandas first it is correctly preserved (I
> guess it's using stored metadata for this).
> I've attached a screenshot showing this behavior.
> As currently there is support for nullable types in pandas, just as in Arrow,
> it would be great to use these types when dealing with columns with null
> values.
> If you are reticent to change this behavior, a param would be nice too (e.g.
> `to_pandas(use_nullable_types: True)`).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
