Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17505 )
Change subject: IMPALA-7501: Slim down partition metadata in LocalCatalog mode ...................................................................... Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@356 PS2, Line 356: 4: bool want_partition_metadata : > nit, move to the end so that field numbers are in sequence? Done http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@397 PS2, Line 397: : // Set if 'want_partition_names' was set in TTableInfoSelector. : 2: optional string name : : // Set if 'want_hms_partition' was set in TTableInfoSelector. : 3: optional hive_metastore.Partition hms_partition : > can we move them to the end so that field numbers are in sequence. Done http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@357 PS4, Line 357: > IMO, it is less error-prone to keep this field after field 11. You can add Done http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@397 PS4, Line 397: : // Set if 'want_partition_names' was set in TTableInfoSelector. : 2: optional string name : : // Set if 'want_hms_partition' was set in TTableInfoSelector. : 3: optional hive_metastore.Partition hms_partition : > Can we move them to after line 428. Otherwise it may become error-prone if Done http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@960 PS4, Line 960: hdfsStorageDescriptor Yeah, the optimization is borrown from the legacy catalog mode. > Is it possible to send only one copy of it at the table level and then > construct it from the table's storageDescriptor like what catalogd does? No, HdfsStorageDescriptor contains the file format and dilimiters, and we do support partition-level file formats. Here's an example from https://impala.apache.org/docs/build/html/topics/impala_alter_table.html alter table fast_growing_data partition (year = 2016, month in (10,11,12)) set fileformat parquet; I think we also support partition specifit dilimiters for text partitions. -- To view, visit http://gerrit.cloudera.org:8080/17505 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I307e7a8193b54a7b3ab93d9ebd194766bbdbd977 Gerrit-Change-Number: 17505 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Wed, 02 Jun 2021 01:20:00 +0000 Gerrit-HasComments: Yes
