yyanyy opened a new pull request, #59662: URL: https://github.com/apache/spark/pull/59662
### What changes were proposed in this pull request? Add the evolving `ScanBuilderInfo` API with an `isInSubquery()` flag and a default `SupportsRead.newScanBuilder(ScanBuilderInfo, CaseInsensitiveStringMap)` overload. Spark supplies this context when creating batch scan builders, including the destination context when rebuilding merged scans. The default overload delegates to the existing options-only method, preserving compatibility with existing connectors and passing the original read options unchanged. Each builder receives a fresh context object. The context is advisory and must not change the rows returned by a scan. ### Why are the changes needed? The open-source Delta connector needs to avoid expensive transaction-log replay for statistics collection during subquery planning. Subsequent planning steps may revisit the same tables for more thorough statistics analysis, so collecting those statistics early can add avoidable planning overhead. This requirement surfaced while migrating the Delta connector to DataSource V2 (DSV2). Preserving its existing behavior requires a way to distinguish subquery planning from regular planning. The subquery flag provides that context to the connector so it can choose the appropriate statistics effort when creating a scan builder. This also creates a foundation for Spark to provide connectors with additional query-planning signals in the future. Those signals could describe where Spark is in planning a query, helping connectors choose the appropriate work at each stage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Passed 199 tests across `V2ScanRelationPushDownSubquerySuite`, `MergeSubplansSuite`, and `DataSourceV2Suite`: ```text build/sbt 'sql/testOnly *V2ScanRelationPushDownSubquerySuite *MergeSubplansSuite org.apache.spark.sql.connector.DataSourceV2Suite' ``` The new coverage checks context availability before pushdown and statistics selection, compatibility with options-only connectors, unchanged option-map identity and contents, fresh context across repeated table use, scalar/IN/EXISTS subqueries, local/global temporary-view reuse and replacement, and the destination context of rebuilt merged scans. Java Checkstyle, Scala style checks for SQL production and test sources, and `git diff --check` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex 0.163.0 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
