Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13224 )
Change subject: IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution ...................................................................... Patch Set 7: I ran this (without the other patches on top) and it used a bunch of memory and overloaded my machine (which has 4 cores and 32GB of mem). Various things ran out of memory and the OS was killing stuff. Have you seen anything like this? One thing I think we should look at is how many YARN nodes we run. Currently, we run one YARN resource manager and three YARN node managers. I'm not sure that having three YARN node managers is useful. I tried this patch with one YARN node manager, and my machine was less on fire (though memory usage still made it up to 28GB a few times). There might be an execution time penalty though, so it could be that other ways of decreasing resource usage would make more sense. (Specifically, I changed this line: https://github.com/apache/impala/blob/master/testdata/cluster/admin#L238-L239 to also remove yarn-nodemanager from all but the first node). -- To view, visit http://gerrit.cloudera.org:8080/13224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d Gerrit-Change-Number: 13224 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yongzhi Chen <[email protected]> Gerrit-Comment-Date: Tue, 07 May 2019 22:00:42 +0000 Gerrit-HasComments: No
