Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24587
Change subject: IMPALA-15185: Increase Hive query parallelism from 2 to 8 ...................................................................... IMPALA-15185: Increase Hive query parallelism from 2 to 8 The combination of Tarn/Tez/Hive configs limited the number of parallel Hive queries to 2, which slowed down Hive heavy tests and the dataload. Specifically each Hive query needs a Yarn application master (AM), and mem settings limited AM count to 2. This change tweaks some configs. The math on typical Impala test job (m6i-4xl): total mem: 64 GB Yarn mem: 44.0 GB (unchanged) AM mem%: 10% -> 20% AM mem: 4.4 GB -> 8.8 GB mem/AM: 2.0 GB -> 1.0 GB max AMs: 2 -> 8 worker mem: 39.4 GB -> 35.2 GB mem/worker: 2 GB (unchanged) max workers: 19 -> 17 worker/AM: 9 -> 2 The main goal is to speed up dataload, which is Hive heavy and tries to do multiple jobs in parallel. ACID compactions also seem to become much faster (>1m -> 10-20s), not sure why. 2 worker/AM looks small, but this is a worst (or best...) case scenario where 8 Hive queries do heavy work in parallel. Most tables are small and should be fine with 1 worker. Hive queries during tests are usually minimal (like insterting 1 row). If the change brings instability, then the number of AMs can be reduced. Change-Id: I0b17c99b4f925bcce4604f8e595de33bd87236c4 --- M fe/src/test/resources/hive-site.xml.py M testdata/cluster/node_templates/common/etc/hadoop/conf/capacity-scheduler.xml M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/24587/1 -- To view, visit http://gerrit.cloudera.org:8080/24587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0b17c99b4f925bcce4604f8e595de33bd87236c4 Gerrit-Change-Number: 24587 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
