baumgold opened a new issue #285:
URL: https://github.com/apache/arrow-julia/issues/285
When writing a table with Arrow.jl that contains a nullable column, the
Arrow data cannot be read by Pyarrow. Note that when using Missing instead of
Nothing Pyarrow can read the data written by Arrow.jl.
```julia
julia> Arrow.write("/tmp/foo.arrow",
(col=Vector{Union{Nothing,Int32}}([1,2,3,nothing]),))
"/tmp/foo.arrow"
julia> Arrow.Table("/tmp/foo.arrow")
Arrow.Table with 4 rows, 1 columns, and schema:
:col Union{Missing, Nothing, Int32}
```
```python
In [9]: df = pandas.read_feather("/tmp/foo.arrow")
-----------------------------------------------------------------
ArrowNotImplementedError Traceback (most recent call last)
<ipython-input-9-9b9a515df158> in <module>
----> 1 df = pandas.read_feather("/tmp/foo.arrow")
~/miniconda3/lib/python3.9/site-packages/pandas/io/feather_format.py in
read_feather(path, columns, use_threads, storage_options)
128 ) as handles:
129
--> 130 return feather.read_feather(
131 handles.handle, columns=columns,
use_threads=bool(use_threads)
132 )
~/miniconda3/lib/python3.9/site-packages/pyarrow/feather.py in
read_feather(source, columns, use_threads, memory_map)
218 """
219 _check_pandas_version()
--> 220 return (read_table(source, columns=columns,
memory_map=memory_map)
221 .to_pandas(use_threads=use_threads))
222
~/miniconda3/lib/python3.9/site-packages/pyarrow/array.pxi in
pyarrow.lib._PandasConvertible.to_pandas()
~/miniconda3/lib/python3.9/site-packages/pyarrow/table.pxi in
pyarrow.lib.Table._to_pandas()
~/miniconda3/lib/python3.9/site-packages/pyarrow/pandas_compat.py in
table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)
787 _check_data_column_metadata_consistency(all_columns)
788 columns = _deserialize_column_index(table, all_columns,
column_indexes)
--> 789 blocks = _table_to_blocks(options, table, categories,
ext_columns_dtypes)
790
791 axes = [columns, index]
~/miniconda3/lib/python3.9/site-packages/pyarrow/pandas_compat.py in
_table_to_blocks(options, block_table, categories, extension_columns)
1126 # Convert an arrow table to Block from the internal pandas API
1127 columns = block_table.column_names
-> 1128 result = pa.lib.table_to_blocks(options, block_table, categories,
1129 list(extension_columns.keys()))
1130 return [_reconstruct_block(item, columns, extension_columns)
~/miniconda3/lib/python3.9/site-packages/pyarrow/table.pxi in
pyarrow.lib.table_to_blocks()
~/miniconda3/lib/python3.9/site-packages/pyarrow/error.pxi in
pyarrow.lib.check_status()
ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of
type dense_union<: null=0, : int32 not null=1> is known.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
