GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19615
[SPARK-19611][SQL][followup] set dataSchema correctly in
HiveMetastoreCatalog.convertToLogicalRelation
## What changes were proposed in this pull request?
We made a mistake in https://github.com/apache/spark/pull/16944 . In
`HiveMetastoreCatalog#inferIfNeeded` we infer the data schema, merge with full
schema, and return the new full schema. At caller side we treat the full schema
as data schema and set it to `HadoopFsRelation`.
This doesn't cause any problem because both parquet and orc can work with a
wrong data schema that has extra columns, but it's better to fix this mistake.
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark infer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19615.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19615
----
commit 46f530fe777c921d43a2f323abc91d8bb69423d5
Author: Wenchen Fan <[email protected]>
Date: 2017-10-30T23:05:57Z
set dataSchema correctly in HiveMetastoreCatalog.convertToLogicalRelation
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
