kimtkyeom commented on a change in pull request #27888: [SPARK-31116][SQL]
Consider case sensitivity in ParquetRowConverter
URL: https://github.com/apache/spark/pull/27888#discussion_r391525196
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
##########
@@ -804,6 +804,162 @@ abstract class ParquetQuerySuite extends QueryTest with
ParquetTest with SharedS
}
}
+ test("SPARK-31116: Select simple parquet columns correctly in case
insensitive manner") {
Review comment:
In addition, I noticed that json does not follow case sensitivity even in
spark 2.4.4. Below is my local machine test using spark-shell
```
20/03/12 19:20:19 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
20/03/12 19:20:24 WARN util.Utils: Service 'SparkUI' could not bind on port
4040. Attempting port 4041.
Spark context Web UI available at http://61.75.36.130:4041
Spark context available as 'sc' (master = local[*], app id =
local-1584008425035).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = Seq("A").toDF("camelCase")
df: org.apache.spark.sql.DataFrame = [camelCase: string]
scala> df.write.format("json").save("./json_simple")
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val sch2 = new StructType().add("camelcase", StringType)
sch2: org.apache.spark.sql.types.StructType =
StructType(StructField(camelcase,StringType,true))
scala> spark.read.format("json").schema(sch2).load("./json_simple").show()
+---------+
|camelcase|
+---------+
| null. |
+---------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
