aokolnychyi edited a comment on issue #1485: URL: https://github.com/apache/iceberg/issues/1485#issuecomment-700967918
This is a great discussion, thanks to everyone for taking a look at this! I completely agree with @rdblue and @bryanck that it is up to Spark to invalidate the cached relations just like it did for V1 tables. The only case where we would have to match it in Iceberg is in our stored procedures that may rollback table state or produce new snapshots (we will review this case by case). Then I'll sum up this discussion with the following points (feel free to correct): - Spark should refresh its cache for V2 tables to match V1 behavior (covers REFRESH, INSERT, etc). - Iceberg should refresh Spark cache when modifying tables in procedures. - We should extend `SparkTable` with an ability to refresh the underlying table if Iceberg caching catalog is not used to have consistent behavior independently whether cashing is enabled in Iceberg catalogs. - We need to override `equals` and `hashCode` in our table implementations. - We need to route writes through `IcebergSource` to catalogs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
