Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24587
to look at the new patch set (#2).
Change subject: IMPALA-15185: Increase Hive query parallelism from 2 to 8
......................................................................
IMPALA-15185: Increase Hive query parallelism from 2 to 8
The combination of Yarn/Tez/Hive configs limited the number
of parallel Hive queries to 2, which slowed down Hive heavy
tests and the dataload. Specifically each Hive query needs
a Yarn application master (AM), and mem settings limited AM
count to 2.
This change tweaks some configs. The math on typical Impala
test job (m6i-4xl):
total mem: 64 GB
Yarn mem: 44.0 GB (unchanged)
AM mem%: 10% -> 20%
AM mem: 4.4 GB -> 8.8 GB
mem/AM: 2.0 GB -> 1.0 GB
max AMs: 2 -> 8
worker mem: 39.4 GB -> 35.2 GB
mem/worker: 2 GB (unchanged)
max workers: 19 -> 17
worker/AM: 9 -> 2
The main goal is to speed up dataload, which is Hive heavy and
tries to do multiple jobs in parallel. ACID compactions also
seem to become much faster (>1m -> 10-20s), not sure why.
2 worker/AM looks small, but this is a worst (or best...)
case scenario where 8 Hive queries do heavy work in parallel.
Most tables are small and should be fine with 1 worker. Hive
queries during tests are usually minimal (like insterting 1 row).
If the change brings instability, then the number of AMs can
be reduced.
Change-Id: I0b17c99b4f925bcce4604f8e595de33bd87236c4
---
M fe/src/test/resources/hive-site.xml.py
M testdata/cluster/node_templates/common/etc/hadoop/conf/capacity-scheduler.xml
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
3 files changed, 9 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/24587/2
--
To view, visit http://gerrit.cloudera.org:8080/24587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b17c99b4f925bcce4604f8e595de33bd87236c4
Gerrit-Change-Number: 24587
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
