GitHub user CodingCat opened a pull request:
https://github.com/apache/spark/pull/21757
[SQL][SPARK-24797] respect spark.sql.hive.convertMetastoreOrc/Parquet when
buildâ¦
## What changes were proposed in this pull request?
the current code path ignore the value of
spark.sql.hive.convertMetastoreParquet when building data source table
https://github.com/apache/spark/blob/e0559f238009e02c40f65678fec691c07904e8c0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L263
as a result, even I turned off spark.sql.hive.convertMetastoreParquet,
Spark SQL still uses its own parquet reader to access table instead of delegate
to serder
This PR checks the value of the configuration when building data source
table
## How was this patch tested?
existing test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CodingCat/spark SPARK-24797
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21757.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21757
----
commit a5d72cc2cc77aaaada7d8fab0cfc4a48959b774c
Author: Nan Zhu <nanzhu@...>
Date: 2018-07-13T02:44:25Z
respect respect spark.sql.hive.convertMetastoreOrc/Parquet when build the
data source table
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
