Yang Jie created SPARK-60810:
--------------------------------
Summary: Keep CHAR column statistics through read-side padding
Key: SPARK-60810
URL: https://issues.apache.org/jira/browse/SPARK-60810
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
With the default spark.sql.readSideCharPadding=true, ApplyCharTypePadding wraps
every CHAR column of a scan in Alias(StaticInvoke(CharVarcharCodegenUtils,
"readSidePadding", c, N)). EstimationUtils.getAliasStats only recognizes
Alias(Attribute) and foldable aliases, so the column's statistics are lost at
the first Project. For example, after ANALYZE TABLE ... COMPUTE STATISTICS FOR
COLUMNS on a CHAR(5) column with 10 distinct values in 1000 rows, the Aggregate
of SELECT c, count(*) ... GROUP BY c has no rowCount estimate under CBO, while
the same query on a STRING column estimates 10.
ANALYZE TABLE reads the table through the same padding, so the stored
statistics already describe padded values and can be passed through the alias
unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
