[
https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712382#comment-17712382
]
Willi Raschkowski commented on SPARK-43142:
-------------------------------------------
Here's what's happening: {{ImplicitOperators}} methods like {{asc}} rely on a
call to {{expr}}
[(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L149].
The {{UnresolvedAttribute}} returned by {{.attr}} is implicitly converted to
{{DslAttr}}. But {{DslAttr}} does not implement {{expr}} by returning the
attribute it's already wrapping. Instead, it only implements how to convert the
attribute it's wrapping to a string name
[(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L273-L275].
Returning an attribute for an implicitly wrapped attribute is implemented on
the super class {{ImplicitAttribute}} by creating a new {{UnresolvedAttribute}}
on the string name return by {{DslAttr}} (the method call {{s}},
[Github|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L278-L280]).
The problem is that this string name returned by {{DslAttr}} no longer has the
quotes and thus the new {{UnresolvedAttribute}} parses an unquoted identifier.
{code}
scala> "`col/slash`".attr.name
res1: String = col/slash
{code}
> DSL expressions fail on attribute with special characters
> ---------------------------------------------------------
>
> Key: SPARK-43142
> URL: https://issues.apache.org/jira/browse/SPARK-43142
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Willi Raschkowski
> Priority: Major
>
> Expressions on implicitly converted attributes fail if the attributes have
> names containing special characters. They fail even if the attributes are
> backtick-quoted:
> {code:java}
> scala> import org.apache.spark.sql.catalyst.dsl.expressions._
> import org.apache.spark.sql.catalyst.dsl.expressions._
> scala> "`slashed/col`".attr
> res0: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute =
> 'slashed/col
> scala> "`slashed/col`".attr.asc
> org.apache.spark.sql.catalyst.parser.ParseException:
> mismatched input '/' expecting {<EOF>, '.', '-'}(line 1, pos 7)
> == SQL ==
> slashed/col
> -------^^^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
