rdblue commented on a change in pull request #1801:
URL: https://github.com/apache/iceberg/pull/1801#discussion_r529005850
##########
File path:
spark3/src/main/java/org/apache/iceberg/spark/procedures/BaseProcedure.java
##########
@@ -56,7 +65,9 @@ protected BaseProcedure(TableCatalog tableCatalog) {
T result = func.apply(icebergTable);
- refreshSparkCache(ident, sparkTable);
+ if (refreshSparkCache) {
+ refreshSparkCache(ident, sparkTable);
+ }
Review comment:
I think the cache still needs to be refreshed. A commit will refresh the
table reference that is cached in Iceberg, pulling in all changes that happened
since it was loaded. There could be changes unrelated to the manifest rewrite
that get pulled in and would cause the cache to be stale compared with a new
query. I think it is best to refresh Spark's cache whenever a procedure
modifies or refreshes the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
