Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21782 )
Change subject: IMPALA-12876: Add catalogVersion and loaded timestamp in query profiles ...................................................................... IMPALA-12876: Add catalogVersion and loaded timestamp in query profiles When debugging stale metadata, it'd be helpful to know what catalog version of the tables are used and what's the time when catalogd loads those versions. This patch exposes these info in the query profile for each referenced table. E.g. Original Table Versions: tpch.customer, 2249, 1726052668932, Wed Sep 11 19:04:28 CST 2024 tpch.nation, 2255, 1726052790140, Wed Sep 11 19:06:30 CST 2024 tpch.orders, 2257, 1726052803258, Wed Sep 11 19:06:43 CST 2024 tpch.lineitem, 2254, 1726052785384, Wed Sep 11 19:06:25 CST 2024 tpch.supplier, 2256, 1726052794235, Wed Sep 11 19:06:34 CST 2024 Each line consists of the table name, catalog version, loaded timestamp and the timestamp string. Implementation: The loaded timestamp is updated whenever a CatalogObject updates its catalog version in catalogd. It's passed to impalads with the TCatalogObject broadcasted by statestore, or in DDL/DML responses. Currently, the loaded timestamp is added for table, view, function, data source, and hdfs cache pool in catalogd. However, only those of table and view are applied used in impalad. For the loaded timestamp of other types, users can check them in the /catalog WebUI of catalogd. Tests: - Adds e2e test Change-Id: I94b2fd59ed5aca664d6db4448c61ad21a88a4f98 Reviewed-on: http://gerrit.cloudera.org:8080/21782 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectImpl.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CtasTargetTable.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/VirtualTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/query_test/test_observability.py M tests/shell/test_shell_commandline.py 20 files changed, 150 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I94b2fd59ed5aca664d6db4448c61ad21a88a4f98 Gerrit-Change-Number: 21782 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
