mattaubury opened a new issue, #37111:
URL: https://github.com/apache/arrow/issues/37111
### Describe the bug, including details regarding any error messages,
version, and platform.
Using `pyarrow-12.0.1` on RHEL8 Intel.
dataset filters work when the table is read into memory, but break when the
table is a referenced Parquet file. The following code demonstrates the issue:
```
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
value_type = pa.duration("s")
# Create a file with a single row
table = pa.table([pa.array([1], value_type)], ["t"])
filename = "test.parquet"
pq.write_table(table, filename, version="2.6")
# Filter the row in-memory using dataset, THIS WORKS
table2 = pq.read_table(filename)
dataset = ds.dataset(table2)
x = dataset.to_table(filter=ds.field("t") == pa.scalar(1, type=value_type))
# Filter the row using dataset from disk, THIS BREAKS
dataset = ds.dataset(filename, format="parquet")
y = dataset.to_table(filter=ds.field("t") == pa.scalar(1, type=value_type))
assert x == y
```
The assert is not reached, instead we get the error:
```
Traceback (most recent call last):
File "break_filter.py", line 19, in <module>
y = dataset.to_table(filter=ds.field("t") == pa.scalar(1,
type=value_type))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 546, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3449, in
pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel
matching input types (duration[us], int8)
```
Superficially, it appears as though the duration scalar is being prematurely
unboxed.
The issue is not present with other value_types (such as `int64`,
`timestamp`, `date32`), but a potentially related issue occurs with `value_type
= pa.time32("s")`:
```
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel
matching input types (time32[ms], time32[s])
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
