Solved it.
The anonymous RDDs can be cached in the cacheManager in SQLContext.
In order to remove all the cached content use:
sqlContext.clearCache()
The warning symptom about failed data frame registration is the following
entry in the log:
16/04/16 20:18:39 [tp439928219-110] WARN CacheManager: Asked to cache
already cached data.
Piotr
On Wed, Mar 30, 2016 at 3:10 PM, psmolinski <[email protected]>
wrote:
> bumping up the topic.
>
> For the moment I stay with 1.5.2, but I would like to switch to 1.6.x and
> this issue is a blocker.
>
> Thanks,
> Piotr
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Cached-Parquet-file-paths-problem-tp26576p26637.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
