Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14148
@rxin Currently, we do not run schema inference every time when metadata
cache contains the plan. Based on my understanding, that is the major reason
why we introduced the metadata cache at the very beginning.
I think it is not hard to store the schema of data source tables in the
external catalog (Hive metastore). However, `Refresh Table` only refreshes the
metadata cache and the data cache. It does not update the schema stored in the
external catalog. If we do not store the schema in the external catalog, it
works well. Otherwise, we have to refresh the schema info in the external
catalog.
To implement your idea, I can submit a PR for the release 2.1 tomorrow. We
can discuss it in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
