kbendick edited a comment on issue #1485: URL: https://github.com/apache/iceberg/issues/1485#issuecomment-699567199
> I actually would have assumed it was a bug if a "Cache" command was invalidated by another table operation, in my mind it should snapshot the table state at that moment. I know because the behavior is lazy in Spark your guarantees on "when" are a bit more iffy, but I think the Spark cache shouldn't be automatically invalidated. @RussellSpitzer I'd like to point out that as of Spark 3.0, the behavior of `CACHE TABLE` is eager, unless the DDL is `CACHE LAZY TABLE` [as documented here](https://spark.apache.org/docs/3.0.0/sql-ref-syntax-aux-cache-cache-table.html#syntax) for caching tables (at least non-temporary tables). I understand that the behavior is different depending on whether or not the cached data is a `VIEW` or a `TABLE`, most notably whether or not those cached plans / datasets are considered temporary or not. I went searching for the best documentation I could on this, but the most recent thing I could find was the JIRA ticket that covers the update to allow for `Non-cascading cache invalidation`. It appears that non-cascading cache invalidation _should_ only take place when a temporary dataset or table/view is dropped in order to save memory, but the underlying query plans for the other cached collections of data are still valid (to avoid using `Table` or `View` or `Dataset`). E.g. cache invalidation is not cascaded as of Spark 2.4.x unless the query plans of the downstream cached datasets would have been invalidated, which does not include strictly dropping the original cached view / table / dataset that they might have been derived from. The non-cascading behavior appears to really only cover the cases when data is no longer persisted in memory or on disk etc in order to reclaim resources, but that otherwise `REFRESH`ing temporary tables _should_ invalidate the cache (possibly depen dent on changes, I'm not sure) and any downstream cached views / query plans that would also be invalidated. cc @aokolnychyi. So what I read in this ticket, which might be the most up to date documentation on "intended" cache behavior in various scenarios seems to line up with @rdblue's latest response. The JIRA ticket is here: https://issues.apache.org/jira/browse/SPARK-24596 For convenience and reference, I've quoted the pertinent description of the intended high level behaviors from that JIRA ticket. Note that `regular mode` here refers to cache invalidation that does cascade to derived on cluster caches of datasets / query plans. ``` 1. Drop tables and regular (persistent) views: regular mode2 2. Drop temporary views: non-cascading mode 3. Modify table contents (INSERT/UPDATE/MERGE/DELETE): regular mode 4. Call DataSet.unpersist(): non-cascading mode 5. Call Catalog.uncacheTable(): follow the same convention as drop tables/view, which is, use non-cascading mode for temporary views and regular mode for the rest ``` In the associated PR (which does appear to have been merged in and released as of Spark 2.4.x), the description is further clarified with the following TLDR ([link to the PR])(https://github.com/apache/spark/pull/21594) ``` Note that a regular (persistent) view is a database object just like a table, so after dropping a regular view (whether cached or not cached), any query referring to that view should no long be valid. Hence if a cached persistent view is dropped, we need to invalidate the all dependent caches so that exceptions will be thrown for any later reference. On the other hand, a temporary view is in fact equivalent to an unnamed DataSet, and dropping a temporary view should have no impact on queries referencing that view. Thus we should do non-cascading uncaching for temporary views, which also guarantees a consistent uncaching behavior between temporary views and unnamed DataSets. ``` I know that this discussion is about the observed change in behaviors that occur because of V2 tables (and potentially from using a catalog that is not part of Spark proper), but since this is the most recent explanation I could find that covered the questions that are being discussed, I thought it was relevant to the conversation. After going through all of this, I will say that I had originally agreed with @RussellSpitzer's take, especially on point 1, that the cache is a point in time referece and immutable (throwing away potentially the notion of consistency with `lazy` caching, which is no longer the default). However, our own experience and the documentation I could find does not line up with that. I will say now that I'm mostly in agreement with @rdblue, especially that the (possible) need / desire to call `REFRESH` should be documented and recommended in the near term. I also agree that we should delegate the choosing of cache invalidation for queries to Spark, but ensure that we have a means to get that information to Spark to allow it to do its thing. Particularly at the Catalog level. In the long term, I think Spark users will expect the latest behavior of Spark as the standard. Though how long it takes for changes in behavior to propagate into the minds of users as the latest / most correct is not something we could really estimate. I use Spark quite often (though I've yet to do a production upgrade to Spark 3, admittedly) and I will admit I was surprised by some of the things I learned. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
