Hello Laszlo Gaal, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23092
to look at the new patch set (#8).
Change subject: IMPALA-14697: Generate TPC-H/TPC-DS data in parallel
......................................................................
IMPALA-14697: Generate TPC-H/TPC-DS data in parallel
For performance test jobs, generating the datasets at higher
scales can take significant time. Currently, this is running
a single invocation of the generator binary in a single
thread. Both TPC-H and TPC-DS have an ability to generate data
in parallel, producing multiple files for larger tables. This
keeps the current behavior for scale factor 1 to avoid having
a different number of files. This generates higher scale
factors with $(nproc) parallel threads. The LOAD DATA statements
for these tables already reference the directory name itself,
so it doesn't matter that there are more files.
Testing:
- Ran core job
- Loaded TPC-H scale 42
- Loaded TPC-DS scale 20
Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263
---
M testdata/datasets/tpcds/preload
M testdata/datasets/tpch/preload
2 files changed, 185 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/23092/8
--
To view, visit http://gerrit.cloudera.org:8080/23092
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263
Gerrit-Change-Number: 23092
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Laszlo Gaal <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
