Gengliang Wang created SPARK-61914:
--------------------------------------

             Summary: Add domain evaluation helpers to the Rust SDK for native 
data sources
                 Key: SPARK-61914
                 URL: https://issues.apache.org/jira/browse/SPARK-61914
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


SPARK-61306 (PR #60314) passes the domains of the columns that the filters of a 
query constrain to Rust native data sources, with ScanBuilder::push_domains. 
The SDK exposes Domain, ValueRange and Value, but no helper to evaluate them, 
so every connector that skips data reimplements their semantics: the Spark SQL 
order of values (-0.0 equal to 0.0, NaN only through nan_allowed), 
null_allowed, the nulls of struct fields under a null parent, and the 
conversion of a Value to an Arrow literal of the type of its column. A mistake 
there skips rows that match the filters, which Spark cannot detect, since it 
only filters the rows that the scan returns.

This adds helpers to the SDK, without a change to the C ABI:
- convert a Value to an Arrow scalar of the type of its column;
- evaluate a domain on an Arrow array, as a mask of the rows whose values are 
in it;
- decide, from arrays of the min and max values and the null counts of row 
ranges, such as Parquet row groups or pages, which ranges may have a value in a 
domain. The caller says whether its statistics cover NaN, since Parquet's leave 
it out.

The helpers can be extracted from the Parquet example of PR #59692 (filter.rs 
and prune.rs), which then uses them. A vectorized implementation adds arrow-ord 
as a dependency of the SDK.

Depends on SPARK-61306.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to