Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22184#discussion_r213386126
--- Diff: docs/sql-programming-guide.md ---
@@ -1895,6 +1895,10 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
- Since Spark 2.4, File listing for compute statistics is done in
parallel by default. This can be disabled by setting
`spark.sql.parallelFileListingInStatsComputation.enabled` to `False`.
- Since Spark 2.4, Metadata files (e.g. Parquet summary files) and
temporary files are not counted as data files when calculating table size
during Statistics computation.
+## Upgrading From Spark SQL 2.3.1 to 2.3.2 and above
+
+ - In version 2.3.1 and earlier, when reading from a Parquet table, Spark
always returns null for any column whose column names in Hive metastore schema
and Parquet schema are in different letter cases, no matter whether
`spark.sql.caseSensitive` is set to true or false. Since 2.3.2, when
`spark.sql.caseSensitive` is set to false, Spark does case insensitive column
name resolution between Hive metastore schema and Parquet schema, so even
column names are in different letter cases, Spark returns corresponding column
values. An exception is thrown if there is ambiguity, i.e. more than one
Parquet column is matched.
--- End diff --
> For Spark native parquet tables that were created by us, this is a bug
fix because the previous work does not respect spark.sql.caseSensitive; for the
parquet tables created by Hive, the field resolution should be consistent no
matter whether it is using our reader or Hive parquet reader.
@gatorsmile, need confirm with you, about consistent, we have some kinds of
tables.
1. parquet table created by Spark (using parquet) read by Spark reader
2. parquet table created by Spark (using hive) read by Spark reader
3. parquet table created by Spark (using hive) read by Hive reader
4. parquet table created by Hive read by Spark reader
5. parquet table created by Hive read by Hive reader
Do you want all of them to be consitent? Or 2,3,4,5 consitent is enough?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
