amol- commented on a change in pull request #11076:
URL: https://github.com/apache/arrow/pull/11076#discussion_r702060847
##########
File path: python/pyarrow/tests/parquet/test_dataset.py
##########
@@ -505,11 +505,19 @@ def test_filters_invalid_pred_op(tempdir,
use_legacy_dataset):
use_legacy_dataset=use_legacy_dataset)
assert dataset.read().num_rows == 0
- with pytest.raises(ValueError):
- pq.ParquetDataset(base_path,
- filesystem=fs,
- filters=[('integers', '!=', {3})],
- use_legacy_dataset=use_legacy_dataset)
+ if use_legacy_dataset:
+ with pytest.raises(ValueError):
+ pq.ParquetDataset(base_path,
+ filesystem=fs,
+ filters=[('integers', '!=', {3})],
+ use_legacy_dataset=use_legacy_dataset)
+ else:
+ dataset = pq.ParquetDataset(base_path,
+ filesystem=fs,
+ filters=[('integers', '!=', {3})],
+ use_legacy_dataset=use_legacy_dataset)
+ with pytest.raises(NotImplementedError):
+ assert dataset.read().num_rows == 0
Review comment:
`X != {3}` isn't supported for legacy datasets because they have an
explicit check when compiling the expression. They don't support _any_
container object.
```
elif not isinstance(f_value, str) and isinstance(f_value, Collection):
raise ValueError(
"Op '%s' not supported with a collection value", op)
```
for new datasets api instead it was a side effect of the fact we didn't
support `sets` when converting scalar values. If instead of `X != {3}` we had
`X != [3]` that would have raised for the legacy datasets but would have
compiled ok for new datasets.
As now the expression compiles fine for new dataset the error changes to the
fact that the `not_equal` function has no kernel that supports a `list` as the
argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
