Hello Tianyi Wang, Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11561
to look at the new patch set (#5).
Change subject: IMPALA-7626: Throttle catalog partial RPC requests
......................................................................
IMPALA-7626: Throttle catalog partial RPC requests
With more coordinators running in local catalog mode, the expected RPC
traffic on the Catalog server is higher compared to the non-local-catalog
mode. Each such RPC is handled in its own thread and consumes some
non-trivial CPU for serializing and deserializing the metadata.
With this change, the maximum number of threads performing the actual work are
capped to a certain limit at any point and the remaining requests (if any)
are blocked until the current requests are serviced or they they exceed
the configured timeout and abort. Adds the following parameters for controlling
this behavior.
--catalog_max_parallel_partial_rpc
--catalog_partial_rpc_queue_timeout_s
--catalog_partial_rpc_queue_timeout_s controls the timeout for queued
requests.
Added some basic supportability to examine the queue length via metrics.
Added a unit test to make sure the concurrent requests for this RPC
method does not exceed the configured value.
Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
13 files changed, 265 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/11561/5
--
To view, visit http://gerrit.cloudera.org:8080/11561
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142
Gerrit-Change-Number: 11561
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tianyi Wang <[email protected]>
Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
