[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. IMPALA-10343: increase control_service_queue_mem_limit --control_service_queue_mem_limit is set to 1% by default to increase the maximum size of the queue in typical production deployments. E.g. an Impala daemon with a 50GB memory limit will have a limit of 512MB on the control service queue. Add --control_service_queue_mem_limit_floor_bytes so that this does not have the unintended effect of reducing the memory given to the control service queue. I.e. the default behaviour does not change for impala daemons with a daemon mem limit of <= 5000MB, but does increase the control service queue memory limit for impala daemons with mem limits > 5000MB. The default process memory limit in the mocked backend test ExecEnv is changed to be 8GB. Previously it was unlimited, so we couldn't calculate 1% of it. It cannot be unlimited in an actual impalad since IMPALA-5653 was fixed. Testing: This had been previously problematic on a 64 node TPC-DS workload with mt_dop=12 where impalads had ~100GB of memory. Status report RPCs would fail and have to be retried. We tested this new value on the same workload and the retries were avoided. Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Reviewed-on: http://gerrit.cloudera.org:8080/16848 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Tauber-Marshall --- M be/src/runtime/test-env.h M be/src/service/control-service.cc M tests/custom_cluster/test_rpc_timeout.py 3 files changed, 12 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Tauber-Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 17:56:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 11:51:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7818/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 06:39:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Some backend tests were failing because the process mem limit was initialised to -1. The fix was pretty easy, but not completely trivial so would be worth looking at -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 06:19:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6750/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 06:18:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Hello Thomas Tauber-Marshall, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16848 to look at the new patch set (#3). Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. IMPALA-10343: increase control_service_queue_mem_limit --control_service_queue_mem_limit is set to 1% by default to increase the maximum size of the queue in typical production deployments. E.g. an Impala daemon with a 50GB memory limit will have a limit of 512MB on the control service queue. Add --control_service_queue_mem_limit_floor_bytes so that this does not have the unintended effect of reducing the memory given to the control service queue. I.e. the default behaviour does not change for impala daemons with a daemon mem limit of <= 5000MB, but does increase the control service queue memory limit for impala daemons with mem limits > 5000MB. The default process memory limit in the mocked backend test ExecEnv is changed to be 8GB. Previously it was unlimited, so we couldn't calculate 1% of it. It cannot be unlimited in an actual impalad since IMPALA-5653 was fixed. Testing: This had been previously problematic on a 64 node TPC-DS workload with mt_dop=12 where impalads had ~100GB of memory. Status report RPCs would fail and have to be retried. We tested this new value on the same workload and the retries were avoided. Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa --- M be/src/runtime/test-env.h M be/src/service/control-service.cc M tests/custom_cluster/test_rpc_timeout.py 3 files changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/16848/3 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6748/ -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 04:54:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Dec 2020 23:32:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6748/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Dec 2020 23:19:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 2: Code-Review+2 carry -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Dec 2020 23:19:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Hello Thomas Tauber-Marshall, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16848 to look at the new patch set (#2). Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. IMPALA-10343: increase control_service_queue_mem_limit --control_service_queue_mem_limit is set to 1% by default to increase the maximum size of the queue in typical production deployments. E.g. an Impala daemon with a 50GB memory limit will have a limit of 512MB on the control service queue. Add --control_service_queue_mem_limit_floor_bytes so that this does not have the unintended effect of reducing the memory given to the control service queue. I.e. the default behaviour does not change for impala daemons with a daemon mem limit of <= 5000MB, but does increase the control service queue memory limit for impala daemons with mem limits > 5000MB. Testing: This had been previously problematic on a 64 node TPC-DS workload with mt_dop=12 where impalads had ~100GB of memory. Status report RPCs would fail and have to be retried. We tested this new value on the same workload and the retries were avoided. Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa --- M be/src/service/control-service.cc M tests/custom_cluster/test_rpc_timeout.py 2 files changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/16848/2 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16848/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16848/1//COMMIT_MSG@19 PS1, Line 19: <= > should be >= ? I think I got it right - I mean that if an Impalad has --mem_limit=5000mb or less, then the control service queue is 50MB so this patch does not change anything. I think this was pretty unclear so I rewrote this. -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Dec 2020 23:19:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16848/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16848/1//COMMIT_MSG@19 PS1, Line 19: <= should be >= ? -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Dec 2020 23:03:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 09 Dec 2020 22:59:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7813/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 09 Dec 2020 22:55:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16848 Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. IMPALA-10343: increase control_service_queue_mem_limit --control_service_queue_mem_limit is set to 1% by default to increase the maximum size of the queue in typical production deployments. E.g. an Impala daemon with a 50GB memory limit will have a limit of 512MB on the control service queue. Add --control_service_queue_mem_limit_floor_bytes so that this does not have the unintended effect of reducing the memory given to the queue. If the impala daemon has a mem limit of <= 5000MB, this patch has no effect to default behaviour. Testing: This had been previously problematic on a 64 node TPC-DS workload with mt_dop=12 where impalads had ~100GB of memory. Status report RPCs would fail and have to be retried. We tested this new value on the same workload and the retries were avoided. Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa --- M be/src/service/control-service.cc M tests/custom_cluster/test_rpc_timeout.py 2 files changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/16848/1 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
