GitHub user chenghao-intel opened a pull request:
https://github.com/apache/spark/pull/8023
[SPARK-9689][SQL]Fix bug of not invalidate the cache for
InsertIntoHadoopFsRelation
We didn't refresh the cache in `InsertIntoHadoopFsRelation`, however, even
by adding the fresh operation, I also noticed that the spark plan is immutable,
this is probably a bug if the underlying files changes (added/deleted).
So I mark the the PhyscialRdd as a mutable, and always create the new RDD
while `doExecute()` function called.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chenghao-intel/spark cache_refresh
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8023.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8023
----
commit 0fcbda24efe84ea02163f47dd2d61b477f3de69f
Author: Cheng Hao <[email protected]>
Date: 2015-08-06T15:52:09Z
Fix bug of invalidate the cache for HadoopFsRelation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
