rjzamora opened a new issue, #34504: URL: https://github.com/apache/arrow/issues/34504
### Describe the enhancement requested As discussed in https://github.com/dask/dask/issues/9845, it is not currently possible to pass a `List[Tuple]`-formatted filter to `filters_to_expression` that can be converted into an expression that will filter null/nan values correctly. This is because something like `[("a", "!=", np.nan)` will result in `<pyarrow.compute.Expression (a != nan)>` instead of `<pyarrow.compute.Expression invert(is_null(a, {nan_is_null=true}))>`. @j-bennet suggested that we add support for `"is"` and `"is not"` operators for this, and @jorisvandenbossche seemed receptive to this idea. Since this was a blocking issue for some RAPIDS users, dask has temporarily added its own [custom implementation of `filters_to_expression`](https://github.com/dask/dask/blob/main/dask/dataframe/io/parquet/arrow.py#L338-L407). Is there any interest in adopting a similar version of this utility in pyarrow? **Note**: It would be nice if the `filters_to_expression` utility made it possible to mimic pandas behavior and avoid null propagation for predicates like `[("a", "!=", 1)]` (by taking such a predicate to mean `<pyarrow.compute.Expression (is_null(a, {nan_is_null=true}) or (a != 1))>` on the arrow side). Dask's custom implementation adds the `propagate_null` argument for this purpose. ### Component(s) Parquet, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
