jpivarski opened a new issue, #41308:
URL: https://github.com/apache/arrow/issues/41308
### Describe the bug, including details regarding any error messages,
version, and platform.
The pyarrow constructor prevents such a type:
```python
>>> import pyarrow as pa
>>> pa.field("name", pa.null(), nullable=False) # not allowed
```
raises
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/types.pxi", line 2266, in pyarrow.lib.field
ValueError: A null type field may not be non-nullable
```
but there's a way around it, perhaps unintended:
```python
>>> pa.field("name", pa.null()).with_nullable(False) # equivalent, but
allowed
pyarrow.Field<name: null not null>
```
On the other hand, data created with this type can't be written to Parquet:
```
...
File
"/home/jpivarski/irishep/awkward/src/awkward/operations/ak_to_parquet.py", line
416, in _impl
with pyarrow_parquet.ParquetWriter(
File
"/home/jpivarski/mambaforge/lib/python3.10/site-packages/pyarrow/parquet/core.py",
line 1010, in __init__
self.writer = _parquet.ParquetWriter(
File "pyarrow/_parquet.pyx", line 2157, in
pyarrow._parquet.ParquetWriter.__cinit__
File "pyarrow/error.pxi", line 154, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: NullType Arrow field must be nullable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jpivarski/irishep/awkward/src/awkward/_dispatch.py", line 38,
in dispatch
with OperationErrorContext(name, args, kwargs):
File "/home/jpivarski/irishep/awkward/src/awkward/_errors.py", line 85, in
__exit__
self.handle_exception(exception_type, exception_value)
File "/home/jpivarski/irishep/awkward/src/awkward/_errors.py", line 95, in
handle_exception
raise self.decorate_exception(cls, exception)
pyarrow.lib.ArrowInvalid: NullType Arrow field must be nullable
```
On the other, other hand, the specification is silent on the issue:
https://arrow.apache.org/docs/format/Columnar.html#null-layout
This came up in the context of Awkward Array conversions into Arrow and
Parquet (through pyarrow), scikit-hep/awkward#2340. We can make this
possibly-invalid Arrow type with
```python
>>> import awkward as ak
>>> arrow_array1 = ak.to_arrow(ak.Array([]), extensionarray=False,
list_to32=True)
>>> arrow_array1
<pyarrow.lib.NullArray object at 0x7bbd7a80f340>
0 nulls
>>> arrow_array1.type
DataType(null)
```
and
```python
>>> arrow_array2 = ak.to_arrow(ak.Array([[], [], []]), extensionarray=False,
list_to32=True)
>>> arrow_array2
<pyarrow.lib.ListArray object at 0x7bbd7a80e920>
[
0 nulls,
0 nulls,
0 nulls
]
>>> arrow_array2.type
ListType(list<item: null not null>)
```
but get the stack trace above if we try to convert these arrays into
Parquet. (Awkward Arrays default to non-nullable, and the list array conversion
to Arrow uses `with_nullable`, so we may have found this unintended back-door
to creating the non-nullable null type.)
Right now, we're developing a work-around to avoid creating Arrow arrays
with non-nullable null types, assuming that it is invalid, as the error
messages say. But...
**Is it invalid?** If so, the `with_nullable` back-door should be closed,
and this fact should be documented in the specification.
**Is it valid?** If so, then the normal constructor should be able to
construct it, and it should be writable to Parquet.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
