yyanyy opened a new pull request, #57487:
URL: https://github.com/apache/spark/pull/57487
Refactor how a DataSource V2 (DSv2) scan reports statistics, aligning it with
the v1 (`LogicalRelation` / `CatalogStatistics.toPlanStats`) path.
- `computeStats` gates on `cboEnabled || planStatsEnabled`: full statistics
only when needed, otherwise a cheaper size-only estimate via the new
`SupportsReportStatistics.estimateSizeInBytes()` default method.
- New `SupportsReportStatistics.reflectsFullyPushedDownFilters()` default
method
(`true`). When a connector returns `false`, `V2ScanRelationPushDown`
re-adds
the fully pushed, deterministic filters above the scan so
`FilterEstimation`
adjusts the stats. The rewrite is row-equivalent, idempotent, and
best-effort
under column pruning (pruned-reference predicates are dropped).
- `transformV2Stats` infers `sizeInBytes` from the row count when no size is
reported (matching v1); the dead `defaultRowCount` parameter is removed.
- `V1ScanWrapper` implements `SupportsReportStatistics`, delegating to the
wrapped v1 scan so its stats are no longer dropped.
- Shared logic extracted into `V2StatisticsUtils` (null-safe).
Both new methods have defaults that preserve existing behavior, so existing
connectors need no change.
### Why are the changes needed?
To let DSv2 connectors produce statistics more cheaply and let Spark plan on
more accurate statistics, matching v1:
- Cheaper when only size is needed: most queries (CBO/plan stats off) only
need
`sizeInBytes`, so a connector can skip the expensive row/column statistics.
- Accurate after pushdown: whole-table stats are adjusted by
`FilterEstimation`
instead of planning on too-large estimates, improving plan choices.
- Fuller use of reported stats: v1 scan stats and a row-count-derived size
are
now used instead of the default size, at no extra cost to the connector.
### Does this PR introduce _any_ user-facing change?
New API introduced but no behavior change by default except for the
following cases:
- when a source reports a row count but no size, the size is inferred from
the row count (previously the
default size was used)
- a v1 scan implementing `SupportsReportStatistics` now has its reported
statistics surfaced (previously dropped).
Both changes make the estimates more accurate and align with the v1 path.
### How was this patch tested?
New unit and end-to-end tests in `DataSourceV2RelationSuite` and
`DataSourceV2Suite` cover: the CBO / plan-stats gating, size inference
(including the zero-row case), null/empty statistics handling,
`V1ScanWrapper`
delegation, and the post-pushdown filter re-adding (including the recursion
through stacked residual filters).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
