[
https://issues.apache.org/jira/browse/ARROW-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931820#comment-16931820
]
Wes McKinney commented on ARROW-6529:
-------------------------------------
The reason is that the Feather writer is "faking" a NULL type by writing a UTF8
column with all nulls
https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.cc#L615
In both R and Python this results in an all-null array when read back.
Probably the easiest quick way to fix this would be to write "user_metadata"
for the Column to signal to C++ that the output should be NullArray. Longer
term we need to deprecate the details in feather.fbs and make Feather a wrapper
around the standard IPC protocol which is much better supported and featureful
I don't think this is urgent for 0.15.0 so I'm removing from the milestone
> [C++] Feather: slow writing of NullArray
> ----------------------------------------
>
> Key: ARROW-6529
> URL: https://issues.apache.org/jira/browse/ARROW-6529
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Joris Van den Bossche
> Assignee: Wes McKinney
> Priority: Major
> Labels: feather
> Fix For: 1.0.0
>
>
> From
> https://stackoverflow.com/questions/57877017/pandas-feather-format-is-slow-when-writing-a-column-of-none
> Smaller example with just using pyarrow, it seems that writing an array of
> nulls takes much longer than an array of for example ints, which seems a bit
> strange:
> {code}
> In [93]: arr = pa.array([None]*1000, type='int64')
> In [94]: %%timeit
> ...: w = pyarrow.feather.FeatherWriter('__test.feather')
> ...: w.writer.write_array('x', arr)
> ...: w.writer.close()
> 31.4 µs ± 464 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> In [95]: arr = pa.array([None]*1000)
> In [96]: arr
> Out[96]:
> <pyarrow.lib.NullArray object at 0x7fa47a23ca40>
> 1000 nulls
> In [97]: %%timeit
> ...: w = pyarrow.feather.FeatherWriter('__test.feather')
> ...: w.writer.write_array('x', arr)
> ...: w.writer.close()
> 3.75 ms ± 64.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
> {code}
> So writing the same length NullArray takes ca 100x more time compared to an
> array of nulls but with Integer type.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
