Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21402 )
Change subject: IMPALA-12712: Invalidate metadata on table should set better createEventId ...................................................................... Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/21402/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21402/1//COMMIT_MSG@13 PS1, Line 13: always removing the table. Sequence of drop table + create table + > when is the table removed? in case of drop table event? Yeah while processing drop table event. http://gerrit.cloudera.org:8080/#/c/21402/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21402/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6999 PS1, Line 6999: try (MetaStoreClient msClient = catalog_.getMetaStoreClient(catalogTimeline)) { > This is a huge improvement compared to the original state, but there is sti Yeah, your understanding is correct. Unfortunately, currently, there is no way to deal with the above-mentioned scenario. http://gerrit.cloudera.org:8080/#/c/21402/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7044 PS1, Line 7044: } catch (IcebergTableLoadingException e) { : updatedThriftTable = catalog_.invalidateTable(req.getTable_name(), : tblWasRemoved, dbWasAdded, catalogTimeline, eventId); > This is not really related to the current change, but this Iceberg specific My understanding is that, when a refresh on the iceberg table fails we just invalidate the iceberg table. Since the manifest file locations can change from time-to-time, there might be issues with finding the right file block locations for iceberg tables. http://gerrit.cloudera.org:8080/#/c/21402/1/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21402/1/tests/custom_cluster/test_events_custom_configs.py@1271 PS1, Line 1271: self.client.execute("drop table {}.{}".format(unique_database, test_tbl)) : self.run_stmt_in_hive( > About doing these in Hive vs Impala: the issue would also come if the drop if we were to drop the table in hive (since event processor is not synced up) and create table in impala fails with table already exists exception. Also, invalidating table after it is created in impala doesn't fulfill our goal. On the other hand, I have verified that dropping and creating in hive and then doing invalidate will not drop the table in event processor and this test passes. http://gerrit.cloudera.org:8080/#/c/21402/1/tests/custom_cluster/test_events_custom_configs.py@1273 PS1, Line 1273: "create table {}.{} (i int, j int)".format(unique_database, test_tbl)) > nit: +2 indentation Done -- To view, visit http://gerrit.cloudera.org:8080/21402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff6ac18fe8d9e7b25cc41c7e41eecde251fbccdd Gerrit-Change-Number: 21402 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Wed, 22 May 2024 23:01:47 +0000 Gerrit-HasComments: Yes
