Jason Sachs created ARROW-10498:
-----------------------------------
Summary: pyarrow.Table.from_* methods appear to cut off binary
data after an embedded zero byte
Key: ARROW-10498
URL: https://issues.apache.org/jira/browse/ARROW-10498
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 2.0.0
Environment: > python
Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] ::
Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
Reporter: Jason Sachs
The pyarrow.Table.from_* methods appear to cut off binary data after an
embedded zero byte.
{code}
>>> import numpy as np
>>> import pyarrow as pa
>>>
>>> data = np.array([b'', b'', b'', b'Foo!!', b'Bar!!',
... b'\x00Baz!', b'half\x00baked', b''], dtype='|S13')
>>> t = pa.Table.from_pydict({'data':data})
>>> t.to_pandas()
data
0 b''
1 b''
2 b''
3 b'Foo!!'
4 b'Bar!!'
5 b''
6 b'half'
7 b''
>>> import pandas as pd
>>> pd.DataFrame(data)
0
0 b''
1 b''
2 b''
3 b'Foo!!'
4 b'Bar!!'
5 b'\x00Baz!'
6 b'half\x00baked'
7 b''
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
