Gengliang Wang created SPARK-61900:
--------------------------------------

             Summary: Push the domains of filtered columns down to the 
streaming reads of native data sources
                 Key: SPARK-61900
                 URL: https://issues.apache.org/jira/browse/SPARK-61900
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Spark pushes the domains of the filtered columns down to native data sources 
({{push_domains}}, SPARK-61200) only for batch scans. A micro-batch read pushes 
nothing down but the columns to read, which {{create_stream_reader}} gets: 
{{ColumnarScanBuilder}} only derives domains for its batch planner, and the 
table of a native data source does not declare the capability with which a Data 
Source V2 streaming source accepts the filters of a streaming query 
(SPARK-60926).

Proposal: pass the domains to the stream reader too, for example in fields 
appended to {{SparkDsStreamReaderInfo}}, for the data sources that declare a 
capability for it, and expose them in the Rust SDK (SPARK-61282, SPARK-61306). 
As for batch scans, Spark still evaluates all the filters after reading.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to