Thomas Blauth created ARROW-12336:
-------------------------------------
Summary: [C++][Python] Empty Int64 array is of wrong size
Key: ARROW-12336
URL: https://issues.apache.org/jira/browse/ARROW-12336
Project: Apache Arrow
Issue Type: Bug
Components: C++, Python
Environment: macOS 10.15.7
Arrow version: 3.1.0.dev578
Reporter: Thomas Blauth
Setup:
Table with Int64 and str columns; generated using the dataset api; filtered on
str column.
Bug Description:
Calling {{table.to_pandas()}} fails due to an empty array of the ChunkedArray
of the Int64 column. This empty array has a size of 4 Byte when using the arrow
nightly builds and 0 Byte when using arrow 3.0.0.
Note: The bug does not occur when the table only contains an Int64 column.
Minimal example:
{code:python}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet
import pyarrow.dataset
print("Arrow version: " + str(pa.__version__))
print("---------------")
# Only Int64 works fine
df = pd.DataFrame({"Int_col": [1, 2, 10]}, dtype="Int64")
table = pa.table(df)
path_0 = "./test_0.parquet"
pa.parquet.write_table(table, path_0)
schema = pa.parquet.read_schema(path_0)
ds = pa.dataset.FileSystemDataset.from_paths(
paths=[path_0],
filesystem=pa.fs.LocalFileSystem(),
schema=schema,
format=pa.dataset.ParquetFileFormat(),
)
table = ds.to_table(filter=(pa.dataset.field("Int_col") == 3))
print("Size of array: " + str(table.column(0).nbytes))
df = table.to_pandas()
print("---------------")
# Int64 and str crashes
df = pd.DataFrame({"Int_col": [1, 2, 10], "str_col": ["A", "B", "Z"]})
df = df.astype({"Int_col": "Int64"})
table = pa.table(df)
path_1 = "./test_1.parquet"
pa.parquet.write_table(table, path_1)
schema = pa.parquet.read_schema(path_1)
ds = pa.dataset.FileSystemDataset.from_paths(
paths=[path_1],
filesystem=pa.fs.LocalFileSystem(),
schema=schema,
format=pa.dataset.ParquetFileFormat(),
)
table = ds.to_table(filter=(pa.dataset.field("str_col") == "C"))
print("Size of array: " + str(table.column(0).nbytes))
df = table.to_pandas()
{code}
Output :
{code:bash}
Arrow version: 3.1.0.dev578
---------------
Size of array: 0
---------------
Size of array: 4
Traceback (most recent call last):
File "/Users/xxx/empty_array_buffer_size.py", line 47, in <module>
df = table.to_pandas()
File "pyarrow/array.pxi", line 756, in
pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 1740, in pyarrow.lib.Table._to_pandas
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 794, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 1135, in _table_to_blocks
return [_reconstruct_block(item, columns, extension_columns)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 1135, in <listcomp>
return [_reconstruct_block(item, columns, extension_columns)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 753, in _reconstruct_block
pd_ext_arr = pandas_dtype.__from_arrow__(arr)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pandas/core/arrays/integer.py",
line 117, in __from_arrow__
data, mask = pyarrow_array_to_numpy_and_mask(arr, dtype=self.type)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pandas/core/arrays/_arrow_utils.py",
line 32, in pyarrow_array_to_numpy_and_mask
data = np.frombuffer(buflist[1], dtype=dtype)[arr.offset : arr.offset +
len(arr)]
ValueError: buffer size must be a multiple of element size
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
