ArvidJB opened a new issue, #36308:
URL: https://github.com/apache/arrow/issues/36308
### Describe the bug, including details regarding any error messages,
version, and platform.
When converting a numpy array of bytes (dtype='S3' for example) pyarrow
seems to make all values the same fixed-length and pad with null characters:
```
In [2]: pa.__version__
Out[2]: '12.0.1'
In [3]: a = np.array([b'a', b'ab', b'abc'])
In [4]: a
Out[4]: array([b'a', b'ab', b'abc'], dtype='|S3')
In [5]: b = pa.array(a, type=pa.string())
In [6]: b
Out[6]:
<pyarrow.lib.StringArray object at 0x7fb4490083a0>
[
"a",
"ab",
"abc"
]
In [7]: [x.as_py() for x in b]
Out[7]: ['a\x00\x00', 'ab\x00', 'abc']
```
I don't think this is the intended behavior here?
Note that the stringification of `StringArray` hides this issue since it
truncates after encountering the first null character. To be observe it you
need to convert to Python strings.
This also means that the output of `to_numpy` is not the same as the
original array:
```
In [11]: b.to_numpy(zero_copy_only=False)
Out[11]: array(['a\x00\x00', 'ab\x00', 'abc'], dtype=object)
```
If you agree that this is a bug I am happy to contribute a patch.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
