[
https://issues.apache.org/jira/browse/SPARK-60810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-60810:
-----------------------------
Affects Version/s: 4.1.4
4.0.5
4.2.0
3.5.8
3.4.0
4.3.0
> Keep CHAR column statistics through read-side padding
> -----------------------------------------------------
>
> Key: SPARK-60810
> URL: https://issues.apache.org/jira/browse/SPARK-60810
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0, 3.5.8, 4.2.0, 4.3.0, 5.0.0, 4.0.5, 4.1.4
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> With the default spark.sql.readSideCharPadding=true, ApplyCharTypePadding
> wraps every CHAR column of a scan in
> Alias(StaticInvoke(CharVarcharCodegenUtils, "readSidePadding", c, N)).
> EstimationUtils.getAliasStats only recognizes Alias(Attribute) and foldable
> aliases, so the column's statistics are lost at the first Project. For
> example, after ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS on a CHAR(5)
> column with 10 distinct values in 1000 rows, the Aggregate of SELECT c,
> count(*) ... GROUP BY c has no rowCount estimate under CBO, while the same
> query on a STRING column estimates 10.
> ANALYZE TABLE reads the table through the same padding, so the stored
> statistics already describe padded values and can be passed through the alias
> unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
