Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/23852 )
Change subject: IMPALA-14661: Optimize admissiond memory usage by compressing exec requests ...................................................................... IMPALA-14661: Optimize admissiond memory usage by compressing exec requests In global admissiond, the TQueryExecRequest can be very large for complex queries, consuming large memory while queries are queued. This patch adds support for compressing TQueryExecRequest when sending it to the admission control service through AdmitQuery RPC. This reduces memory usage in admissiond for large query execution requests. Compression uses LZ4 and is optional, enabled by the new added admission_control_rpc_compress starting flag. The uncompressed path remains unchanged. Adds a new TQueryExecRequestCompressed thrift struct along with compression and decompression helper functions. The admission controller now handles both compressed and uncompressed requests through a common AdmissionExecRequest abstraction. Compressed requests are decompressed lazily and cached to reduce memory. Decompression timing is carefully controlled. Requests are initially decompressed at submission, but if a request is queued, the decompressed request cache is released to reduce memory usage. When a queued request is later dequeued, it is decompressed again and the decompressed cache is retained. Since admission uses FIFO ordering, a dequeued request is expected to be at the head of the queue and may be accessed multiple times if not admitted immediately. Retaining the cache in this case avoids repeated decompression. This patch also removes the query_options field in AdmissionRequest to eliminate ambiguity between TExecRequest.query_options and the query options nested in TQueryExecRequest. ClientRequestState is updated to sync the top-level TExecRequest.query_options into the nested TQueryExecRequest before admission. As a result, the admission controller now reads query options from TQueryExecRequest, enforcing a single source of truth for admission logic. Adds admissiond metrics to track compressed size, uncompressed size, and compression ratio for query execution requests. Testing: Adds unit tests for Thrift compression and decompression helpers. Updates admission controller tests to cover compressed requests. Passed exhaustive tests. Change-Id: I5a676d1a806451cbf84b0a3f8a706d7c6655e12d --- M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.h M be/src/scheduling/admission-control-service.cc M be/src/scheduling/admission-control-service.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/remote-admission-control-client.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/protobuf/admission_control_service.proto M common/thrift/Query.thrift M common/thrift/metrics.json M tests/custom_cluster/test_admission_controller.py 14 files changed, 419 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/23852/2 -- To view, visit http://gerrit.cloudera.org:8080/23852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5a676d1a806451cbf84b0a3f8a706d7c6655e12d Gerrit-Change-Number: 23852 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
