Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 )
Change subject: IMPALA-7322: Add storage wait time to profile ...................................................................... Patch Set 3: Code-Review+1 (1 comment) Just a comment on the commit message. I'm not super familiar with this code, so would be good if @Bharath could take a look as well http://gerrit.cloudera.org:8080/#/c/13786/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13786/3//COMMIT_MSG@26 PS3, Line 26: Sample output: : Profile for Catalog V1: (storage-load-time is the added property and : it is part of Metadata load in Query Compilation): : After ran a hbase query (Metadata load finished is divided into : several lines because of limitation of commit message): : Query Compilation: 4s401ms : - Metadata load started: 661.084us (661.084us) : - Metadata load finished. loaded-tables=1/1 : load-requests=1 catalog-updates=3 : storage-load-time=233ms: 3s819ms (3s819ms) : - Analysis finished: 3s820ms (763.979us) : - Value transfer graph computed: 3s820ms (63.193us) : Profile for Catalog V2: (StorageLoad.Time is the added property and it : is in CatalogFetch): : Frontend: : - CatalogFetch.ColumnStats.Misses: 1 : - CatalogFetch.ColumnStats.Requests: 1 : - CatalogFetch.ColumnStats.Time: 0 : - CatalogFetch.Config.Misses: 1 : - CatalogFetch.Config.Requests: 1 : - CatalogFetch.Config.Time: 3ms : - CatalogFetch.DatabaseList.Hits: 1 : - CatalogFetch.DatabaseList.Requests: 1 : - CatalogFetch.DatabaseList.Time: 0 : - CatalogFetch.PartitionLists.Misses: 1 : - CatalogFetch.PartitionLists.Requests: 1 : - CatalogFetch.PartitionLists.Time: 4ms : - CatalogFetch.Partitions.Hits: 2 : - CatalogFetch.Partitions.Misses: 1 : - CatalogFetch.Partitions.Requests: 3 : - CatalogFetch.Partitions.Time: 1ms : - CatalogFetch.RPCs.Bytes: 1.01 KB (1036) : - CatalogFetch.RPCs.Requests: 4 : - CatalogFetch.RPCs.Time: 93ms : - CatalogFetch.StorageLoad.Time: 68ms : - CatalogFetch.TableNames.Hits: 2 : - CatalogFetch.TableNames.Requests: 2 : - CatalogFetch.TableNames.Time: 0 : - CatalogFetch.Tables.Misses: 1 : - CatalogFetch.Tables.Requests: 1 : - CatalogFetch.Tables.Time: 91ms : Catalog metrics(this sample is from a hdfs table): : storage-metadata-load-duration: : Count: 1 : Mean rate: 0.0085 : 1 min. rate: 0.032 : 5 min. rate: 0.1386 : 15 min. rate: 0.177 : Min (msec): 111 : Max (msec): 111 : Mean (msec): 111.1802 : Median (msec): 111.1802 : 75th-% (msec): 111.1802 : 95th-% (msec): 111.1802 : 99th-% (msec): 111.1802 I still find this hard to understand, would recommend re-formatting as follows: Sample Output: Profile for Catalog V1 ('storage-load-time' is the added property and it is part of the 'Metadata load finished' section of the profile under the 'Query Compilation' section. The example below was taken after running a HBase query. 'Metadata load finished' is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Profile for Catalog V2 ('StorageLoad.Time' is the added property and it is part of the 'CatalogFetch' counters): Frontend: - CatalogFetch.ColumnStats.Misses: 1 - CatalogFetch.ColumnStats.Requests: 1 - CatalogFetch.ColumnStats.Time: 0 - CatalogFetch.Config.Misses: 1 - CatalogFetch.Config.Requests: 1 - CatalogFetch.Config.Time: 3ms - CatalogFetch.DatabaseList.Hits: 1 - CatalogFetch.DatabaseList.Requests: 1 - CatalogFetch.DatabaseList.Time: 0 - CatalogFetch.PartitionLists.Misses: 1 - CatalogFetch.PartitionLists.Requests: 1 - CatalogFetch.PartitionLists.Time: 4ms - CatalogFetch.Partitions.Hits: 2 - CatalogFetch.Partitions.Misses: 1 - CatalogFetch.Partitions.Requests: 3 - CatalogFetch.Partitions.Time: 1ms - CatalogFetch.RPCs.Bytes: 1.01 KB (1036) - CatalogFetch.RPCs.Requests: 4 - CatalogFetch.RPCs.Time: 93ms - CatalogFetch.StorageLoad.Time: 68ms - CatalogFetch.TableNames.Hits: 2 - CatalogFetch.TableNames.Requests: 2 - CatalogFetch.TableNames.Time: 0 - CatalogFetch.Tables.Misses: 1 - CatalogFetch.Tables.Requests: 1 - CatalogFetch.Tables.Time: 91ms Catalog metrics (this sample is from a HDFS table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen <yc...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Yongzhi Chen <yc...@cloudera.com> Gerrit-Comment-Date: Tue, 10 Sep 2019 22:36:09 +0000 Gerrit-HasComments: Yes
