Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17505 )
Change subject: IMPALA-7501: Slim down partition metadata in LocalCatalog mode ...................................................................... Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@356 PS2, Line 356: // ... each such partition should include the HMS partition struct. : 12: bool want_hms_partition nit, move to the end so that field numbers are in sequence? http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@397 PS2, Line 397: // Fields 10-13 are set if 'want_partition_metadata' was set in TTableInfoSelector. : // These fields are actual info of hms_partition that Impala needs, and are better : // compressed. : 10: optional map<string, string> hms_parameters : 11: optional i64 write_id : 12: optional CatalogObjects.THdfsStorageDescriptor hdfs_storage_descriptor : 13: optional CatalogObjects.THdfsPartitionLocation location can we move them to the end so that field numbers are in sequence. http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@357 PS4, Line 357: 12: bool want_hms_partition IMO, it is less error-prone to keep this field after field 11. You can add a comment about the difference between want_partition_metadata and want_hms_partition to make it more readable after moving this line below. http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@397 PS4, Line 397: // Fields 10-13 are set if 'want_partition_metadata' was set in TTableInfoSelector. : // These fields are actual info of hms_partition that Impala needs, and are better : // compressed. : 10: optional map<string, string> hms_parameters : 11: optional i64 write_id : 12: optional CatalogObjects.THdfsStorageDescriptor hdfs_storage_descriptor : 13: optional CatalogObjects.THdfsPartitionLocation location Can we move them to after line 428. Otherwise it may become error-prone if someone adds a new field to this struct and reuses the field identifiers. http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@960 PS4, Line 960: hdfsStorageDescriptor It looks like we are sending the HdfsStorageDescriptor for each partition here. We just changed it into a more optimized form. Is it possible to send only one copy of it at the table level and then construct it from the table's storageDescriptor like what catalogd does? -- To view, visit http://gerrit.cloudera.org:8080/17505 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I307e7a8193b54a7b3ab93d9ebd194766bbdbd977 Gerrit-Change-Number: 17505 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Tue, 01 Jun 2021 23:26:52 +0000 Gerrit-HasComments: Yes
