[ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712382#comment-17712382 ]
Willi Raschkowski commented on SPARK-43142: ------------------------------------------- Here's what's happening: {{ImplicitOperators}} methods like {{asc}} rely on a call to {{expr}} [(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L149]. The {{UnresolvedAttribute}} returned by {{.attr}} is implicitly converted to {{DslAttr}}. But {{DslAttr}} does not implement {{expr}} by returning the attribute it's already wrapping. Instead, it only implements how to convert the attribute it's wrapping to a string name [(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L273-L275]. Returning an attribute for an implicitly wrapped attribute is implemented on the super class {{ImplicitAttribute}} by creating a new {{UnresolvedAttribute}} on the string name return by {{DslAttr}} (the method call {{s}}, [Github|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L278-L280]). The problem is that this string name returned by {{DslAttr}} no longer has the quotes and thus the new {{UnresolvedAttribute}} parses an unquoted identifier. {code} scala> "`col/slash`".attr.name res1: String = col/slash {code} > DSL expressions fail on attribute with special characters > --------------------------------------------------------- > > Key: SPARK-43142 > URL: https://issues.apache.org/jira/browse/SPARK-43142 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.4.0 > Reporter: Willi Raschkowski > Priority: Major > > Expressions on implicitly converted attributes fail if the attributes have > names containing special characters. They fail even if the attributes are > backtick-quoted: > {code:java} > scala> import org.apache.spark.sql.catalyst.dsl.expressions._ > import org.apache.spark.sql.catalyst.dsl.expressions._ > scala> "`slashed/col`".attr > res0: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = > 'slashed/col > scala> "`slashed/col`".attr.asc > org.apache.spark.sql.catalyst.parser.ParseException: > mismatched input '/' expecting {<EOF>, '.', '-'}(line 1, pos 7) > == SQL == > slashed/col > -------^^^ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org