GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19163
[SPARK-21936][SQL][2.2] backward compatibility test framework for
HiveExternalCatalog
backport https://github.com/apache/spark/pull/19148 to 2.2
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19163.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19163
----
commit 7178ccfbd726149354ad2f9544ed978537efe70f
Author: Wenchen Fan <[email protected]>
Date: 2017-09-08T06:21:49Z
[SPARK-21936][SQL] backward compatibility test framework for
HiveExternalCatalog
`HiveExternalCatalog` is a semi-public interface. When creating tables,
`HiveExternalCatalog` converts the table metadata to hive table format and save
into hive metastore. It's very import to guarantee backward compatibility here,
i.e., tables created by previous Spark versions should still be readable in
newer Spark versions.
Previously we find backward compatibility issues manually, which is really
easy to miss bugs. This PR introduces a test framework to automatically test
`HiveExternalCatalog` backward compatibility, by downloading Spark binaries
with different versions, and create tables with these Spark versions, and read
these tables with current Spark version.
test-only change
Author: Wenchen Fan <[email protected]>
Closes #19148 from cloud-fan/test.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
