Gengliang Wang created SPARK-61902:
--------------------------------------
Summary: Pass the domains of collated strings and of nullable
struct fields to native data sources
Key: SPARK-61902
URL: https://issues.apache.org/jira/browse/SPARK-61902
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
{{push_domains}} (SPARK-61200) gets no domain for some of the columns that the
filters of a query constrain:
* a string column with a non-binary collation, such as {{UTF8_LCASE}}, gets no
domain;
* a field of a struct only gets a domain without null, since statistics
computed from the child array of a struct may not count the rows whose struct
is null as nulls.
The header ({{spark_datasource.h}}) already says that a later version of Spark
may pass domains for other columns, and that a reader must ignore a domain that
it cannot use.
Proposal: define these domains and pass them too, for example the domain of a
collated string to the data sources that declare that they compare strings with
that collation, and expose them in the Rust SDK (SPARK-61282, SPARK-61306).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]