Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 )
Change subject: IMPALA-5737: Tighten minicluster memory limit ...................................................................... Patch Set 1: > Thanks for working on this. > > From a flaky test perspective, I worry that this kind of approach > is too clever. For the in-repo tests, we want to assert that they > have some requirements in terms of memory, and, if those > requirements are met, they always work. To me, the easiest way to > make that happen is always have consistent memory limits, and run > within the same constraints (as much as possible) even if we're > running on a machine with a ton of memory. > > In that sense, I think the existing "70% of system memory" for > impalad memlimit is a bit of an anti-pattern for running tests. (I > think it's actually fine for "start-impala-cluster", which is used > not just for tests, but I wish that run-all-tests pinned that > explicitly to X GB, and we always ran with that.) I had a lot of > trouble tuning that when I worked on the test-with-docker stuff, > and I still see failures (both inside of Docker and out) with > random-seeming mem-limit-exceeded errors. > > One other interesting bit is that the amount of memory we use > actually also changes with number of cores, both because "parallel" > tests scale that way and because there are parallel things going on > inside of Impala. > > Perhaps a simpler approach would be to simply have variables for > all the daemons, and have a "profile" of sorts that tunes them in a > small number of ways. i.e., instead of supporting all possible > memory sizes, we support "I see you have 20GB of RAM" and "I see > you have over 40GB" of RAM, and that reduces the potential for > noise. > > If you do go with this approach, I'd put all the math in one place. > Leave all the scripts to have simple variables like HS2_LIMIT_MB > (i.e, not just Hive, pin it to the specific daemon), and do all the > math centrally. I think it'll be easier to follow. The profile idea sounds good. I think I'll go with that. -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tianyi Wang <[email protected]> Gerrit-Comment-Date: Wed, 02 May 2018 03:28:57 +0000 Gerrit-HasComments: No
