Gengliang Wang created SPARK-61900:
--------------------------------------
Summary: Push the domains of filtered columns down to the
streaming reads of native data sources
Key: SPARK-61900
URL: https://issues.apache.org/jira/browse/SPARK-61900
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Spark pushes the domains of the filtered columns down to native data sources
({{push_domains}}, SPARK-61200) only for batch scans. A micro-batch read pushes
nothing down but the columns to read, which {{create_stream_reader}} gets:
{{ColumnarScanBuilder}} only derives domains for its batch planner, and the
table of a native data source does not declare the capability with which a Data
Source V2 streaming source accepts the filters of a streaming query
(SPARK-60926).
Proposal: pass the domains to the stream reader too, for example in fields
appended to {{SparkDsStreamReaderInfo}}, for the data sources that declare a
capability for it, and expose them in the Rust SDK (SPARK-61282, SPARK-61306).
As for batch scans, Spark still evaluates all the filters after reading.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]