Csaba Ringhofer created IMPALA-15185:
----------------------------------------
Summary: Increase Hive parallelism during dataload
Key: IMPALA-15185
URL: https://issues.apache.org/jira/browse/IMPALA-15185
Project: IMPALA
Issue Type: Improvement
Reporter: Csaba Ringhofer
The combination of different configs limit the number of parallel Hive queries
to 2:
{code}
'yarn.scheduler.minimum-allocation-mb': 2048
{code}
{code}
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
{code}
{code}
ret = min(max(available_ram_gb * 1024 - 20 * 1024, 4096), 48 * 1024)
print("Configuring Yarn NM to use {0}MB RAM".format(ret), file=sys.stderr)
{code}
On typical 64GB machines:
44GB for Yarn
4.4GB for AMs (0.1: 10% of total)
2*2GB (min 2GB for each yarn container)
40 GB (rest of yarn mem) for Tez / MR workers
The typical queries during dataload are pretty small and don't need 20 workers
per AM.
The current setup also seems to play a role in the very slow Hive ACID
compactions (don't understand how exactly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
