Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11182 )
Change subject: IMPALA-7436: initial fetch-from-catalogd implementation ...................................................................... Patch Set 4: (14 comments) I did another round. Don't have any major comments. Will take another look after Todd's rebase. http://gerrit.cloudera.org:8080/#/c/11182/4/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/11182/4/be/src/catalog/catalog-server.cc@130 PS4, Line 130: // wait times, etc. Also a TODO about limiting the number of such concurrent RPCs. http://gerrit.cloudera.org:8080/#/c/11182/1/be/src/exec/catalog-op-executor.cc File be/src/exec/catalog-op-executor.cc: http://gerrit.cloudera.org:8080/#/c/11182/1/be/src/exec/catalog-op-executor.cc@289 PS1, Line 289: const TNetworkAddress& address = > Actually this CHECK causes problems for FE tests, where it's somewhat incon I think you could do something like DCHECK(use_local_catalog || TestInfo:: is_test()) if you still want to retain it. http://gerrit.cloudera.org:8080/#/c/11182/4/be/src/service/fe-support.cc File be/src/service/fe-support.cc: http://gerrit.cloudera.org:8080/#/c/11182/4/be/src/service/fe-support.cc@543 PS4, Line 543: nit: spacing. http://gerrit.cloudera.org:8080/#/c/11182/1/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/11182/1/common/thrift/CatalogService.thrift@246 PS1, Line 246: lector. > I think that could be confusing since we usually use Request/Response speci Don't feel too strongly about this. http://gerrit.cloudera.org:8080/#/c/11182/1/common/thrift/CatalogService.thrift@313 PS1, Line 313: A catalog object descriptor: a TCatalogObject with the object name and type fields : // set. : 3: required CatalogObjects.TCatalogOb > In the case of a bad status, we might not be able to send back an object ve Ah gotcha. http://gerrit.cloudera.org:8080/#/c/11182/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/11182/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1828 PS1, Line 1828: public TGetPartialCatalogObjectResponse getPartialCatalogObject( > I'd rather not complicate the inheritance hierarchy here -- I think the cod Fair point. http://gerrit.cloudera.org:8080/#/c/11182/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1879 PS1, Line 1879: dbCache_ > Not according to getAllTableNames() elsewhere in this class -- I think the Ok, the version snapshot consistency is probably important for statestore based transmission since it relies on that property. Agree it does not make sense here and dbCache_ is atomic. http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1827 PS4, Line 1827: nit: Add method docs for these. http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1868 PS4, Line 1868: nit: remove \n http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/Db.java File fe/src/main/java/org/apache/impala/catalog/Db.java: http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/Db.java@443 PS4, Line 443: public TGetPartialCatalogObjectResponse getPartialInfo( nit: method doc. http://gerrit.cloudera.org:8080/#/c/11182/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/11182/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@101 PS1, Line 101: "Catalog object %s changed version from %s to %s while fetching metadata", > Per the commit message and also I think in the javadoc of this exception ty Missed that, thanks for clarifying. http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@263 PS4, Line 263: fds.add(fd.cloneWithNewHostIndex(resp.table_info.network_addresses, hostIndex)); May be add a comment about the lifecycle of hostIndex and fd remapping. I had to spend some time on this to get it right. http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@300 PS4, Line 300: I think it is nice to have some comments bout PartitionRefImp differences between this and directMetadata provider. You added some context in the commit, but I think we should have something in the code too. Basically how each of these implementations distinguish references based on Id/name http://gerrit.cloudera.org:8080/#/c/11182/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@344 PS4, Line 344: nit: remove -- To view, visit http://gerrit.cloudera.org:8080/11182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If49207fc592b1cc552fbcc7199568b6833f86901 Gerrit-Change-Number: 11182 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tianyi Wang <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vuk Ercegovac <[email protected]> Gerrit-Comment-Date: Wed, 15 Aug 2018 00:19:44 +0000 Gerrit-HasComments: Yes
