GitHub user petermaxlee opened a pull request:
https://github.com/apache/spark/pull/13989
[SPARK-16311][SQL] Improve metadata refresh
## What changes were proposed in this pull request?
This patch implements the 3 things specified in SPARK-16311:
(1) Append a message to the FileNotFoundException and say that a workaround
is to do explicitly metadata refresh.
(2) Make metadata refresh work on temporary tables/views.
(3) Make metadata refresh work on Datasets/DataFrames, by introducing a
Dataset.refresh() method.
And one additional small change:
(4) Merge invalidateTable and refreshTable.
## How was this patch tested?
Created a new test suite that creates a temporary directory and then
deletes a file from it to verify Spark can read the directory once refresh is
called.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/petermaxlee/spark SPARK-16311
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13989.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13989
----
commit cbfbbc7d27ae086805625fa41dbcbad50783fee8
Author: petermaxlee <[email protected]>
Date: 2016-06-30T04:50:37Z
[SPARK-16311][SQL] Improve metadata refresh
commit f7150345245accd0e71a351e9da9ebac9b80a520
Author: petermaxlee <[email protected]>
Date: 2016-06-30T04:53:58Z
Add test suite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
