rjzamora opened a new issue, #34504:
URL: https://github.com/apache/arrow/issues/34504

   ### Describe the enhancement requested
   
   As discussed in https://github.com/dask/dask/issues/9845, it is not 
currently possible to pass a `List[Tuple]`-formatted filter to 
`filters_to_expression` that can be converted into an expression that will 
filter null/nan values correctly. This is because something like `[("a", "!=", 
np.nan)` will result in `<pyarrow.compute.Expression (a != nan)>` instead of 
`<pyarrow.compute.Expression invert(is_null(a, {nan_is_null=true}))>`.
   
   @j-bennet suggested that we add support for `"is"` and `"is not"` operators 
for this, and @jorisvandenbossche seemed receptive to this idea.
   
   Since this was a blocking issue for some RAPIDS users, dask has temporarily 
added its own [custom implementation of 
`filters_to_expression`](https://github.com/dask/dask/blob/main/dask/dataframe/io/parquet/arrow.py#L338-L407).
  Is there any interest in adopting a similar version of this utility in 
pyarrow?
   
   **Note**: It would be nice if the `filters_to_expression` utility made it 
possible to mimic pandas behavior and avoid null propagation for predicates 
like `[("a", "!=", 1)]` (by taking such a predicate to mean 
`<pyarrow.compute.Expression (is_null(a, {nan_is_null=true}) or (a != 1))>` on 
the arrow side). Dask's custom implementation adds the `propagate_null` 
argument for this purpose.
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to