Joe McDonnell has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/23092 )
Change subject: IMPALA-14697: Generate TPC-H/TPC-DS data in parallel ...................................................................... IMPALA-14697: Generate TPC-H/TPC-DS data in parallel For performance test jobs, generating the datasets at higher scales can take significant time. Currently, this is running a single invocation of the generator binary in a single thread. Both TPC-H and TPC-DS have an ability to generate data in parallel, producing multiple files for larger tables. This keeps the current behavior for scale factor 1 to avoid having a different number of files. This generates higher scale factors with $(nproc) parallel threads. The LOAD DATA statements for these tables already reference the directory name itself, so it doesn't matter that there are more files. Testing: - Ran core job - Loaded TPC-H scale 42 - Loaded TPC-DS scale 20 Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263 --- M testdata/datasets/tpcds/preload M testdata/datasets/tpch/preload 2 files changed, 184 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/23092/6 -- To view, visit http://gerrit.cloudera.org:8080/23092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263 Gerrit-Change-Number: 23092 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
