Yan Yan created SPARK-60558:
-------------------------------
Summary: Expose subquery planning context to DataSource V2 scan
builders
Key: SPARK-60558
URL: https://issues.apache.org/jira/browse/SPARK-60558
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yan Yan
DataSource V2 connectors can perform expensive statistics computation while
creating and planning a scan. Spark currently does not tell an ordinary batch
table reader whether its scan builder is being created within a subquery, so
the connector cannot use that scope to decide how much statistics work to
perform.
Add a ScanContext argument to a new SupportsRead.newScanBuilder overload and
provide the current subquery planning scope when Spark constructs the builder.
The default overload delegates to the existing options-only method for
compatibility with existing connectors. A connector can use the context to
avoid expensive statistics work while planning a subquery, without changing the
rows returned by its scan.
The context is advisory and describes the scope at builder creation. Later
optimizer rewrites may move or reuse the scan in another scope; a newly
constructed builder receives a new context for its destination scope. This is
not a guarantee about the final query shape.
This change covers ordinary batch reads. Context overloads for row-level
operations and changelog reads can be considered separately when needed.
PR: https://github.com/apache/spark/pull/59662
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
