[
https://issues.apache.org/jira/browse/PHOENIX-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606479#comment-17606479
]
Attila Zsolt Piros edited comment on PHOENIX-6668 at 9/19/22 8:35 AM:
----------------------------------------------------------------------
[~stoty] Thanks for explaining the purpose of the quotes here.
I have run some more tests and I think "spark.sql.caseSensitive=true" is
working but in that case in the given schema all the column names must match
without quotes, so please try to run it this way:
{noformat}
...
val schema = StructType(Seq(
StructField("ID", LongType, nullable = false),
StructField("TABLE1_ID", LongType, nullable = true),
StructField("t2col1", StringType, nullable = true)
))
spark.sqlContext.sql("set spark.sql.caseSensitive=true")
val df = spark.sqlContext.createDataFrame(rdd1, schema)
...
{noformat}
was (Author: attilapiros):
[~stoty] Thanks for explaining the purpose of the quotes here.
I have run some more tests and I think "spark.sql.caseSensitive=true" is
working but in that case in the given schema all the column names must match
without quotes, so please try to run this way:
{noformat}
...
val schema = StructType(Seq(
StructField("ID", LongType, nullable = false),
StructField("TABLE1_ID", LongType, nullable = true),
StructField("t2col1", StringType, nullable = true)
))
spark.sqlContext.sql("set spark.sql.caseSensitive=true")
val df = spark.sqlContext.createDataFrame(rdd1, schema)
...
{noformat}
> Spark3 connector cannot distinguish column name cases
> -----------------------------------------------------
>
> Key: PHOENIX-6668
> URL: https://issues.apache.org/jira/browse/PHOENIX-6668
> Project: Phoenix
> Issue Type: Bug
> Reporter: Istvan Toth
> Priority: Major
>
> The Spark2 connector handled lowercase and mixed case column names correctly
> in DataFrame definitions.
> Spark3 only does case-insensitive column resolving, and even
> _spark.sql.caseSensitive_ doesn't seem to do anything, neither backquouting.
> Again, this is not something that can likely be fixes from the Phoenix side
> without changes in Spark, and this ticket is mainy for documenting this
> regression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
