Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 )
Change subject: IMPALA-5737: Tighten minicluster memory limit ...................................................................... Patch Set 1: Thanks for working on this. >From a flaky test perspective, I worry that this kind of approach is too >clever. For the in-repo tests, we want to assert that they have some >requirements in terms of memory, and, if those requirements are met, they >always work. To me, the easiest way to make that happen is always have >consistent memory limits, and run within the same constraints (as much as >possible) even if we're running on a machine with a ton of memory. In that sense, I think the existing "70% of system memory" for impalad memlimit is a bit of an anti-pattern for running tests. (I think it's actually fine for "start-impala-cluster", which is used not just for tests, but I wish that run-all-tests pinned that explicitly to X GB, and we always ran with that.) I had a lot of trouble tuning that when I worked on the test-with-docker stuff, and I still see failures (both inside of Docker and out) with random-seeming mem-limit-exceeded errors. One other interesting bit is that the amount of memory we use actually also changes with number of cores, both because "parallel" tests scale that way and because there are parallel things going on inside of Impala. Perhaps a simpler approach would be to simply have variables for all the daemons, and have a "profile" of sorts that tunes them in a small number of ways. i.e., instead of supporting all possible memory sizes, we support "I see you have 20GB of RAM" and "I see you have over 40GB" of RAM, and that reduces the potential for noise. If you do go with this approach, I'd put all the math in one place. Leave all the scripts to have simple variables like HS2_LIMIT_MB (i.e, not just Hive, pin it to the specific daemon), and do all the math centrally. I think it'll be easier to follow. -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tianyi Wang <[email protected]> Gerrit-Comment-Date: Wed, 02 May 2018 02:24:09 +0000 Gerrit-HasComments: No
