Gengliang Wang created SPARK-61902:
--------------------------------------

             Summary: Pass the domains of collated strings and of nullable 
struct fields to native data sources
                 Key: SPARK-61902
                 URL: https://issues.apache.org/jira/browse/SPARK-61902
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


{{push_domains}} (SPARK-61200) gets no domain for some of the columns that the 
filters of a query constrain:
* a string column with a non-binary collation, such as {{UTF8_LCASE}}, gets no 
domain;
* a field of a struct only gets a domain without null, since statistics 
computed from the child array of a struct may not count the rows whose struct 
is null as nulls.

The header ({{spark_datasource.h}}) already says that a later version of Spark 
may pass domains for other columns, and that a reader must ignore a domain that 
it cannot use.

Proposal: define these domains and pass them too, for example the domain of a 
collated string to the data sources that declare that they compare strings with 
that collation, and expose them in the Rust SDK (SPARK-61282, SPARK-61306).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to