aokolnychyi commented on issue #1485: URL: https://github.com/apache/iceberg/issues/1485#issuecomment-696913004
> I actually would have assumed it was a bug if a "Cache" command was invalidated by another table operation, in my mind it should snapshot the table state at that moment. I know because the behavior is lazy in Spark your guarantees on "when" are a bit more iffy, but I think the Spark cache shouldn't be automatically invalidated. That depends on how we interpret the cache in Spark. I don't see a description how it is supposed to work so I rely on the current behavior as the correct way of doing things. I interpret cache as something specific to Spark, even more, specific to a Spark application. Internally, `CacheManager` stores a full logical plan and if a table that is part of that plan is refreshed, the cache is invalidated (done by `InsertIntoDataSourceCommand`). I believe users are used to this behavior and would be really surprised if V2 tables behave differently. If we modify a table from the same driver and already know the cache entry is no longer valid, it seems reasonable to update it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
