[
https://issues.apache.org/jira/browse/IMPALA-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096910#comment-18096910
]
ASF subversion and git services commented on IMPALA-15185:
----------------------------------------------------------
Commit 525362c780a87da6de0410d905f60f9f8fdc1faf in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=525362c78 ]
IMPALA-15184: Switch tpcds json from bzip to zstd
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to zstd, which did not have its own db.
>From dataload logs before change:
Loading TPC-H data OK (Took: 5 min 39 sec)
Loading TPC-DS data OK (Took: 22 min 33 sec)
Loading functional-query data OK (Took: 30 min 43 sec)
After change:
Loading TPC-H data OK (Took: 5 min 22 sec)
Loading TPC-DS data OK (Took: 13 min 49 sec)
Loading functional-query data OK (Took: 27 min 30 sec)
Total dataload time is decreased from ~35m to ~30m.
The longest leg functional-query also decreased, likely
due to less contention on the max 2 parallel Hive queries
(see IMPALA-15185 for details on Hive parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Reviewed-on: http://gerrit.cloudera.org:8080/24586
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Increase Hive parallelism during dataload
> -----------------------------------------
>
> Key: IMPALA-15185
> URL: https://issues.apache.org/jira/browse/IMPALA-15185
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Csaba Ringhofer
> Priority: Major
>
> The combination of different configs limit the number of parallel Hive
> queries to 2:
> {code}
> 'yarn.scheduler.minimum-allocation-mb': 2048
> {code}
> {code}
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> {code}
> {code}
> ret = min(max(available_ram_gb * 1024 - 20 * 1024, 4096), 48 * 1024)
> print("Configuring Yarn NM to use {0}MB RAM".format(ret), file=sys.stderr)
> {code}
> On typical 64GB machines:
> 44GB for Yarn
> 4.4GB for AMs (0.1: 10% of total)
> 2*2GB (min 2GB for each yarn container)
> 40 GB (rest of yarn mem) for Tez / MR workers
> The typical queries during dataload are pretty small and don't need 20
> workers per AM.
> The current setup also seems to play a role in the very slow Hive ACID
> compactions (don't understand how exactly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
