Gengliang Wang created SPARK-61914:
--------------------------------------
Summary: Add domain evaluation helpers to the Rust SDK for native
data sources
Key: SPARK-61914
URL: https://issues.apache.org/jira/browse/SPARK-61914
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
SPARK-61306 (PR #60314) passes the domains of the columns that the filters of a
query constrain to Rust native data sources, with ScanBuilder::push_domains.
The SDK exposes Domain, ValueRange and Value, but no helper to evaluate them,
so every connector that skips data reimplements their semantics: the Spark SQL
order of values (-0.0 equal to 0.0, NaN only through nan_allowed),
null_allowed, the nulls of struct fields under a null parent, and the
conversion of a Value to an Arrow literal of the type of its column. A mistake
there skips rows that match the filters, which Spark cannot detect, since it
only filters the rows that the scan returns.
This adds helpers to the SDK, without a change to the C ABI:
- convert a Value to an Arrow scalar of the type of its column;
- evaluate a domain on an Arrow array, as a mask of the rows whose values are
in it;
- decide, from arrays of the min and max values and the null counts of row
ranges, such as Parquet row groups or pages, which ranges may have a value in a
domain. The caller says whether its statistics cover NaN, since Parquet's leave
it out.
The helpers can be extracted from the Parquet example of PR #59692 (filter.rs
and prune.rs), which then uses them. A vectorized implementation adds arrow-ord
as a dependency of the SDK.
Depends on SPARK-61306.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]