[
https://issues.apache.org/jira/browse/IMPALA-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096909#comment-18096909
]
ASF subversion and git services commented on IMPALA-15184:
----------------------------------------------------------
Commit 525362c780a87da6de0410d905f60f9f8fdc1faf in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=525362c78 ]
IMPALA-15184: Switch tpcds json from bzip to zstd
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to zstd, which did not have its own db.
>From dataload logs before change:
Loading TPC-H data OK (Took: 5 min 39 sec)
Loading TPC-DS data OK (Took: 22 min 33 sec)
Loading functional-query data OK (Took: 30 min 43 sec)
After change:
Loading TPC-H data OK (Took: 5 min 22 sec)
Loading TPC-DS data OK (Took: 13 min 49 sec)
Loading functional-query data OK (Took: 27 min 30 sec)
Total dataload time is decreased from ~35m to ~30m.
The longest leg functional-query also decreased, likely
due to less contention on the max 2 parallel Hive queries
(see IMPALA-15185 for details on Hive parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Reviewed-on: http://gerrit.cloudera.org:8080/24586
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Avoid loading tpds json bzip
> ----------------------------
>
> Key: IMPALA-15184
> URL: https://issues.apache.org/jira/browse/IMPALA-15184
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Reporter: Csaba Ringhofer
> Priority: Major
>
> tcpds json bzip is one of the long legs during dataload (~780s)
> bzip is very slow an already tested in smaller datasets - moving to another
> compression could make this much faster
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
