[
https://issues.apache.org/jira/browse/IMPALA-14697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094683#comment-18094683
]
ASF subversion and git services commented on IMPALA-14697:
----------------------------------------------------------
Commit 770bf8464416506811af312858268828b0cef5dc in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=770bf8464 ]
IMPALA-14697: Generate TPC-H/TPC-DS data in parallel
For performance test jobs, generating the datasets at higher
scales can take significant time. Currently, this is running
a single invocation of the generator binary in a single
thread. Both TPC-H and TPC-DS have an ability to generate data
in parallel, producing multiple files for larger tables. This
keeps the current behavior for scale factor 1 to avoid having
a different number of files. This generates higher scale
factors with $(nproc) parallel threads. The LOAD DATA statements
for these tables already reference the directory name itself,
so it doesn't matter that there are more files.
Testing:
- Ran core job
- Loaded TPC-H scale 42
- Loaded TPC-DS scale 20
Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263
Reviewed-on: http://gerrit.cloudera.org:8080/23092
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Generate TPC-DS / TPC-H datasets in parallel
> --------------------------------------------
>
> Key: IMPALA-14697
> URL: https://issues.apache.org/jira/browse/IMPALA-14697
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For performance test jobs, generating TPC-H or TPC-DS at higher scales can
> take noticeable time (several minutes). Currently, a single call to the
> generator is producing all the tables in a single thread. This can be
> parallelized by having multiple threads each generate the data for a single
> table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
