Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11561 )
Change subject: IMPALA-7626: Throttle catalog partial RPC requests ...................................................................... IMPALA-7626: Throttle catalog partial RPC requests With more coordinators running in local catalog mode, the expected RPC traffic on the Catalog server is higher compared to the non-local-catalog mode. Each such RPC is handled in its own thread and consumes some non-trivial CPU for serializing and deserializing the metadata. With this change, the maximum number of threads performing the actual work are capped to a certain limit at any point and the remaining requests (if any) are blocked until the current requests are serviced or they they exceed the configured timeout and abort. Adds the following parameters for controlling this behavior. --catalog_max_parallel_partial_fetch_rpc --catalog_partial_fetch_rpc_queue_timeout_s --catalog_partial_fetch_rpc_queue_timeout_s controls the timeout for queued requests. Added some basic supportability to examine the queue length via metrics. Added a unit test to make sure the concurrent requests for this RPC method does not exceed the configured value. Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Reviewed-on: http://gerrit.cloudera.org:8080/11561 Reviewed-by: Vuk Ercegovac <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 13 files changed, 255 insertions(+), 12 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tianyi Wang <[email protected]> Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
