GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/21259
[SPARK-24112][SQL] Add `convertMetastoreTableProperty` conf
## What changes were proposed in this pull request?
In Apache Spark 2.4,
[SPARK-23355](https://issues.apache.org/jira/browse/SPARK-23355) fixes a bug
which ignores table properties during `convertMetastore` for tables created by
`STORED AS ORC/PARQUET`.
For some Parquet tables having table properties like `TBLPROPERTIES
(parquet.compression 'NONE')`, it was ignored by default before Apache Spark
2.4. After upgrading cluster, Spark will write uncompressed file which is
different from Apache Spark 2.3 and old.
In order to provide full backward-compatibility, this introduces additional
configuration `spark.sql.hive.convertMetastoreTableProperty` to restore the
previous behavior by ignoring table properties.
## How was this patch tested?
Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark
SPARK-convertMetastoreTableProperty
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21259
----
commit eaecabc5a59457a4baf84dbb755dd7b876fdb536
Author: Dongjoon Hyun <dongjoon@...>
Date: 2018-04-27T18:10:55Z
[SPARK-24112][SQL] Add `convertMetastoreTableProperty` conf
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
