[
https://issues.apache.org/jira/browse/ARROW-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906041#comment-16906041
]
Joris Van den Bossche commented on ARROW-5566:
----------------------------------------------
[~Igor Yastrebov] that's the expected behaviour of pyarrow, as by default arrow
uses `NaN` only as actual float value and not as missing value indicator. But
given that pandas uses this as missing value indicator, you can specify you
want the "pandas interpretation":
{code}
In [23]: pa.array([np.nan, 'string'])
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
<ipython-input-23-ffae557beaf6> in <module>
----> 1 pa.array([np.nan, 'string'])
~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Could not convert string with type str: tried to convert to double
In [24]: pa.array([np.nan, 'string'], from_pandas=True)
Out[24]:
<pyarrow.lib.StringArray object at 0x7f25b585de60>
[
null,
"string"
]
{code}
In {{pa.array()}} you need to specify this manually (when passing an array or
list), but in other methods like `Table.from_pandas` or when passing a pandas
Series, this is done automatically.
> [Python] Overhaul type unification from Python sequence in
> arrow::py::InferArrowType
> ------------------------------------------------------------------------------------
>
> Key: ARROW-5566
> URL: https://issues.apache.org/jira/browse/ARROW-5566
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Wes McKinney
> Priority: Major
>
> I'm working on ARROW-4324 and there's some technical debt lying in
> arrow/python/inference.cc because the case where NumPy scalars are mixed with
> non-NumPy Python scalar values, all hell breaks loose. In particular, the
> innocuous {{numpy.nan}} is a Python float, not a NumPy float64, so the
> sequence {{[np.float16(1.5), np.nan]}} can be converted incorrectly.
> Part of what's messy is that NumPy dtype unification is split from general
> type unification. This should all be combined together with the NumPy types
> mapping onto an intermediate value (for unification purposes) that then maps
> ultimately onto an Arrow type
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
