Yan Yan created SPARK-58315:
-------------------------------
Summary: Improve DSv2 scan statistics: compute on demand and keep
them accurate after pushdown
Key: SPARK-58315
URL: https://issues.apache.org/jira/browse/SPARK-58315
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Yan Yan
The DSv2 scan statistics path lags the v1 (toPlanStats) path:
1. computeStats always fetches full statistics, even when Spark only needs
sizeInBytes (CBO/plan stats off). No cheap size-only option for connectors.
2. After a filter is fully pushed down, the source may still report whole-table
stats. Spark can't tell, so it plans on the too-large, pre-filter estimates.
3. Statistics from a v1 scan are silently dropped: V1ScanWrapper doesn't
implement SupportsReportStatistics, so computeStats never sees them.
4. When only a row count is reported, Spark uses the default (huge) size instead
of deriving one from the row count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
