Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18059 )
Change subject: [scripts] add hard memory limit for mini-cluster processes ...................................................................... [scripts] add hard memory limit for mini-cluster processes Before this patch, the start_kudu.sh script ran all the Kudu server processes at the same node where every kudu-tserver and kudu-master assumed it was the only Kudu process running there, so it would automatically self-impose the hard limit to be 80% of all the memory available. However, since there may be many of those, the mini-cluster could end up in an OOM condition. This patch addresses the issue, dividing all the available memory at a node between the kudu-tserver processes the script starts, so they all use up to 80% of all available memory (the same threshold is used by the auto-detection logic on the amount of memory to consume for --memory_limit_hard_bytes in Kudu servers). The kudu-master processes are started without imposing the hard memory limit assuming they usually don't consume a lot of memory: that's to allocate more memory to kudu-tserver processes. This script is supposed to cover a limited set of use cases automatically and it's always possible to add flags setting or overriding the limits set by the script for kudu-master and kudu-tserver processes via the -M/--master-flags and -T/--tserver-flags command line options correspondingly. I verified that the OOM condition that I saw when running a mini-cluster in a GCP VM no longer happens with this patch. Change-Id: I4de6c96fb0227554edbd5e69b29840f84e408326 Reviewed-on: http://gerrit.cloudera.org:8080/18059 Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong <[email protected]> Reviewed-by: Attila Bukor <[email protected]> --- M src/kudu/scripts/start_kudu.sh 1 file changed, 50 insertions(+), 8 deletions(-) Approvals: Kudu Jenkins: Verified Andrew Wong: Looks good to me, approved Attila Bukor: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4de6c96fb0227554edbd5e69b29840f84e408326 Gerrit-Change-Number: 18059 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Attila Bukor <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan <[email protected]>
