[ 
https://issues.apache.org/jira/browse/SPARK-60830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60830:
-----------------------------------
    Labels: pull-request-available  (was: )

> Keep CHAR/VARCHAR column statistics through read-side checks under standard 
> semantics
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-60830
>                 URL: https://issues.apache.org/jira/browse/SPARK-60830
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> With spark.sql.charVarchar.standardSemantics.enabled=true, 
> ApplyCharTypePadding wraps the CHAR columns of a scan in 
> charTypeReadSideCheck and the VARCHAR columns in varcharTypeReadSideCheck 
> instead of readSidePadding. CBO estimation does not recognize these calls, so 
> the columns lose their statistics at the first Project and filters on them 
> are estimated with selectivity 1.0. SPARK-60810 handles readSidePadding, the 
> default setup, only.
> ANALYZE TABLE reads the table through the same checks, so the stored 
> statistics describe the values the checks return. The checks need a matcher 
> of their own: V2ExpressionBuilder uses the readSidePadding matcher to compile 
> CHAR_READ_SIDE_PADDING, and that must not match the checks, which trim and 
> fail on over-long values. The length floor that SPARK-60810 applies to padded 
> CHAR values holds for charTypeReadSideCheck but not for 
> varcharTypeReadSideCheck, whose values have at most N characters.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to