Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11561
Change subject: WIP IMPALA-7626: Throttle partial RPC requests ...................................................................... WIP IMPALA-7626: Throttle partial RPC requests Adds the Catalog server side ability to funnel the incoming getPartialCatalogObject() RPCs into a fixed size threadpool. These RPCs are only made from the coordinators running in the local catalog mode (--use_local_catalog=true). With more coordinators running in local catalog mode, the expected RPC traffic on the Catalog server is higher compared to the non-catalog mode. Each such RPC is handled in its own thread and consumes some non-trivial CPU for serializing and deserializing the metadata. With this change, there is only a fixed size threadpool that services these RPC requests and any requests beyond the threadpool size are queued. Adds the following parameters to control the thread pool size and max queue length: --catalog_partial_rpc_threadpool_size --catalog_partial_rpc_max_queue_len Beyond --catalog_partial_rpc_max_queue_len, the RPCs are rejected. Added unit-tests and some basic supportability to examine the queue length via metrics. Marking this as WIP since this probably needs more e-e testing. Suggestions welcome. Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/catalog/CatalogPartialRpcMgr.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/CatalogPartialRpcMgrTest.java 13 files changed, 364 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/11561/2 -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada <[email protected]>
