Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9282 )
Change subject: IMPALA-6116: Bound memory usage of DataStreamSevice's service queue ...................................................................... IMPALA-6116: Bound memory usage of DataStreamSevice's service queue The fix for IMPALA-6193 added a memory tracker for the memory consumed by the payloads in the service queue of DataStreamService. This change extends it by introducing a bound on the memory usage for that service queue. In addition, it deprecates FLAGS_datastream_service_queue_depth and replaces it with FLAGS_datastream_service_queue_mem_limit. These flags only take effect when KRPC is in use and KRPC was never enabled in any previous releases so it seems safe to do this flag replacement. The new flag FLAGS_datastream_service_queue_mem_limit directly dictates the amount of memory which can be consumed by the service queue of DataStreamService. This allows a more direct control over the memory usage of the queue instead of inferring via the number of entries in the queue. The default value of this flag is left at 0, in which case it will be set to 5% of process memory limit. Testing done: exhaustive debug builds. Updated data-stream-test to exercise the case in which the payload is larger than the limit. Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 Reviewed-on: http://gerrit.cloudera.org:8080/9282 Reviewed-by: Michael Ho <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test-base.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/mem-tracker.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M tests/custom_cluster/test_krpc_mem_usage.py 15 files changed, 281 insertions(+), 161 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 Gerrit-Change-Number: 9282 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
