Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13224 )
Change subject: IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution ...................................................................... Patch Set 7: > > Patch Set 7: > > > > I ran this (without the other patches on top) and it used a bunch > of memory and overloaded my machine (which has 4 cores and 32GB of > mem). Various things ran out of memory and the OS was killing > stuff. Have you seen anything like this? > > > > I've been testing on a box with 256G of RAM so haven't run into it. > > > One thing I think we should look at is how many YARN nodes we > run. Currently, we run one YARN resource manager and three YARN > node managers. I'm not sure that having three YARN node managers is > useful. I tried this patch with one YARN node manager, and my > machine was less on fire (though memory usage still made it up to > 28GB a few times). There might be an execution time penalty though, > so it could be that other ways of decreasing resource usage would > make more sense. (Specifically, I changed this line: > https://github.com/apache/impala/blob/master/testdata/cluster/admin#L238-L239 > to also remove yarn-nodemanager from all but the first node). > > That's a good idea. Another thing we could do is configure > yarn.nodemanager.resource.memory-mb in yarn-site.xml. The default > is 16G, so with 3 NMs it would want 48G of RAM. > > Any suggestions of what a reasonable allocation would be? Do you > think we should auto-size based on the available RAM in the > machine? Seems useful to be able to speed up data loading on larger > boxes, but we could be automatic or explicit about that. We want it to be faster on machines with more RAM, because that is a common test configuration. On my box the minicluster including Impalads takes about 10GB on its own. That might increase a bit under load. Maybe round that up to 12GB to be polite, then use everything left. On the other hand, this needs to get in sooner than later. I think the resource usage is the only blocker for this. We could just pick something conservative (one YARN node with 16GB would work for all developers), get this patch in, then tune it later when Hive 3 development is up and running. -- To view, visit http://gerrit.cloudera.org:8080/13224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d Gerrit-Change-Number: 13224 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yongzhi Chen <[email protected]> Gerrit-Comment-Date: Thu, 09 May 2019 16:55:01 +0000 Gerrit-HasComments: No
