Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23092 )
Change subject: IMPALA-14697: Generate TPC-H/TPC-DS data in parallel ...................................................................... IMPALA-14697: Generate TPC-H/TPC-DS data in parallel For performance test jobs, generating the datasets at higher scales can take significant time. Currently, this is running a single invocation of the generator binary in a single thread. Both TPC-H and TPC-DS have an ability to generate data in parallel, producing multiple files for larger tables. This keeps the current behavior for scale factor 1 to avoid having a different number of files. This generates higher scale factors with $(nproc) parallel threads. The LOAD DATA statements for these tables already reference the directory name itself, so it doesn't matter that there are more files. Testing: - Ran core job - Loaded TPC-H scale 42 - Loaded TPC-DS scale 20 Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263 Reviewed-on: http://gerrit.cloudera.org:8080/23092 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M testdata/datasets/tpcds/preload M testdata/datasets/tpch/preload 2 files changed, 185 insertions(+), 19 deletions(-) Approvals: Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/23092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263 Gerrit-Change-Number: 23092 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
