yyanyy commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4032133433
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
val catalystColStat = ColumnStat(distinct, min, max, nullCount,
avgLen, maxLen, histogram)
- output.foreach(attribute => {
- if (attribute.name.equals(key.describe())) {
- colStats = colStats :+ (attribute -> catalystColStat)
+ // Catalyst statistics are keyed by top-level Attribute, so only
single-part references
+ // can be matched to an output column.
+ val fieldNames = key.fieldNames
+ if (fieldNames.length == 1) {
+ val fieldName = fieldNames.head
+ val matches = output.filter(attribute => resolver(attribute.name,
fieldName))
+ // On an ambiguous case-insensitive match (e.g. outputs "id" and
"ID"), require a unique
+ // exact-name match, otherwise skip so CBO is not fed the wrong
column.
+ val matched = matches match {
Review Comment:
One edge case may be worth handling here: this makes the match unique for
each individual statistics key, but it does not prevent multiple keys from
resolving to the same output attribute.
For example, with output `id` and `columnStats()` containing both `id` and
`ID`, case-insensitive resolution accepts both. `AttributeMap` then collapses
the duplicate `exprId`, so the retained statistics depend on the source map's
traversal order.
Should we group matches by output attribute and prefer a unique exact-name
key, or otherwise skip the ambiguous statistics? A regression test for this
inverse many-keys-to-one-output case may also help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
