Hi, I don't think that the issue you describe is specific to Iceberg in a sense that even for Hive tables if you make changes using an engine that doesn't trigger HMS events, one has to issue refresh/invalidate metadata to see the changes reflected in Impala. Could you share what catalog you use for your Iceberg tables? And what tool do you use for data ingestion into these tables? If you use the HMS backed HiveCatalog as a catalog and an engine that triggers HMS notifications, like Spark or Hive then even for Iceberg tables you can avoid executing refresh manually.
Gabor On Wed, Jan 8, 2025 at 1:48 PM Saulius Valatka <saulius...@gmail.com> wrote: > Hi, > > If I understand correctly, once an Iceberg table is mutated outside of > Impala one has to run a refresh or invalidate statement. We noticed that > running refresh on huge tables can take minutes and while that is happening > querying them is blocked. We have large event tables that are being updated > very frequently in real-time, by default we run a refresh after each > update, so effectively this means such tables are un-queryable, as they're > constantly being refreshed. > > Is there something I'm missing? What would the recommendation here be? >
