[
https://issues.apache.org/jira/browse/SPARK-38592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Du updated SPARK-38592:
------------------------------
Description:
Try to modify the data frame to ensure column names have no special characters.
{code:java}
val newdf = df.select(
df.columns.map { columnName =>
df.col(surroundingBackTickedName(columnName)).as(normalizeName(columnName))
}: _*
){code}
*surroundingBackTickedName()* will enclose column name with backticks.
However, col() kept having issue with column name contains back tick because
{*}parseAttributeName{*}() only takes backticks that appear in pair. I am
wondering if there is a workaround
{code:java}
org.apache.spark.sql.AnalysisException: Cannot resolve column name
"`COLNAME`2`" among (COLID, COLNAME!4, COLNAME#6, COLNAME$7, COLNAME%8,
COLNAME'25, COLNAME(11, COLNAME)12, COLNAME*10, COLNAME+16, COLNAME,26,
COLNAME-13, COLNAME/30, COLNAME:24, COLNAME;23, COLNAME<27, COLNAME=15,
COLNAME>29, COLNAME?31, COLNAME@5, COLNAME`2){code}
was:
Try to modify the data frame to ensure column names have no special characters.
{code:java}
val newdf = df.select(
df.columns.map { columnName =>
df.col(surroundingBackTickedName(columnName)).as(normalizeName(columnName))
}: _*
){code}
col() kept having issue with column name contains back tick
{code:java}
org.apache.spark.sql.AnalysisException: Cannot resolve column name
"`COLNAME`2`" among (COLID, COLNAME!4, COLNAME#6, COLNAME$7, COLNAME%8,
COLNAME'25, COLNAME(11, COLNAME)12, COLNAME*10, COLNAME+16, COLNAME,26,
COLNAME-13, COLNAME/30, COLNAME:24, COLNAME;23, COLNAME<27, COLNAME=15,
COLNAME>29, COLNAME?31, COLNAME@5, COLNAME`2){code}
> Column name contains back tick `
> --------------------------------
>
> Key: SPARK-38592
> URL: https://issues.apache.org/jira/browse/SPARK-38592
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Reporter: Dennis Du
> Priority: Major
> Labels: bulk-closed
>
> Try to modify the data frame to ensure column names have no special
> characters.
> {code:java}
> val newdf = df.select(
> df.columns.map { columnName =>
> df.col(surroundingBackTickedName(columnName)).as(normalizeName(columnName))
> }: _*
> ){code}
> *surroundingBackTickedName()* will enclose column name with backticks.
> However, col() kept having issue with column name contains back tick because
> {*}parseAttributeName{*}() only takes backticks that appear in pair. I am
> wondering if there is a workaround
> {code:java}
> org.apache.spark.sql.AnalysisException: Cannot resolve column name
> "`COLNAME`2`" among (COLID, COLNAME!4, COLNAME#6, COLNAME$7, COLNAME%8,
> COLNAME'25, COLNAME(11, COLNAME)12, COLNAME*10, COLNAME+16, COLNAME,26,
> COLNAME-13, COLNAME/30, COLNAME:24, COLNAME;23, COLNAME<27, COLNAME=15,
> COLNAME>29, COLNAME?31, COLNAME@5, COLNAME`2){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
