[
https://issues.apache.org/jira/browse/IMPALA-11607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623594#comment-17623594
]
ASF subversion and git services commented on IMPALA-11607:
----------------------------------------------------------
Commit c12ac6c27b2df1eae693b44c157d65499f491d21 in impala's branch
refs/heads/master from LPL
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c12ac6c27 ]
IMPALA-11607: Support for TPC-H and TPC-DS test datasets stored as Iceberg
tables
Impala 4.1.0 has officially supported Iceberg tables, but we don't have
tools to generate TPC-H and TPC-DS test datasets for Iceberg tables. We
should use load-data.py to generate test datasets, e.g:
"${IMPALA_HOME}/bin/load-data.py -w tpcds -f -s 10
--table_formats=text/none,iceberg/none".
Testing:
- load-data.py -w tpch -s 1 --table_formats=text/none,parquet/none,
iceberg/none -e exhaustive -f
- load-data.py -w tpcds -s 1 --table_formats=text/none,parquet/zstd,
iceberg/zstd -e exhaustive -f
- single_node_perf_run.py --workloads=tpch --scale=30 --iterations=3 --
table_formats=parquet/lz4,iceberg/lz4 --num_impalads=1 --load 3ee3dc847
Workload: TPCH, Scale Factor: 30
Table Format: parquet/lz4/block
+----------+----------------+-----------+
| Query | Time Taken (s) | Client ID |
+----------+----------------+-----------+
| TPCH-Q1 | 7.73 | 1 |
| TPCH-Q1 | 7.50 | 1 |
| TPCH-Q1 | 7.55 | 1 |
+----------+----------------+-----------+
| TPCH-Q10 | 3.65 | 1 |
| TPCH-Q10 | 3.70 | 1 |
| TPCH-Q10 | 3.69 | 1 |
+----------+----------------+-----------+
| TPCH-Q11 | 0.68 | 1 |
| TPCH-Q11 | 0.68 | 1 |
| TPCH-Q11 | 0.67 | 1 |
+----------+----------------+-----------+
| TPCH-Q12 | 1.33 | 1 |
| TPCH-Q12 | 1.08 | 1 |
| TPCH-Q12 | 1.08 | 1 |
+----------+----------------+-----------+
| TPCH-Q13 | 10.63 | 1 |
| TPCH-Q13 | 10.36 | 1 |
| TPCH-Q13 | 10.52 | 1 |
+----------+----------------+-----------+
| TPCH-Q14 | 1.63 | 1 |
| TPCH-Q14 | 1.79 | 1 |
| TPCH-Q14 | 1.63 | 1 |
+----------+----------------+-----------+
| TPCH-Q15 | 1.44 | 1 |
| TPCH-Q15 | 1.54 | 1 |
| TPCH-Q15 | 1.45 | 1 |
+----------+----------------+-----------+
| TPCH-Q16 | 2.35 | 1 |
| TPCH-Q16 | 2.35 | 1 |
| TPCH-Q16 | 2.40 | 1 |
+----------+----------------+-----------+
| TPCH-Q17 | 1.20 | 1 |
| TPCH-Q17 | 1.13 | 1 |
| TPCH-Q17 | 1.10 | 1 |
+----------+----------------+-----------+
| TPCH-Q18 | 17.50 | 1 |
| TPCH-Q18 | 17.04 | 1 |
| TPCH-Q18 | 14.43 | 1 |
+----------+----------------+-----------+
| TPCH-Q19 | 1.61 | 1 |
| TPCH-Q19 | 1.56 | 1 |
| TPCH-Q19 | 1.56 | 1 |
+----------+----------------+-----------+
| TPCH-Q2 | 0.68 | 1 |
| TPCH-Q2 | 0.68 | 1 |
| TPCH-Q2 | 0.68 | 1 |
+----------+----------------+-----------+
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.15 | 1 |
+----------+----------------+-----------+
| TPCH-Q21 | 24.94 | 1 |
| TPCH-Q21 | 26.42 | 1 |
| TPCH-Q21 | 25.62 | 1 |
+----------+----------------+-----------+
| TPCH-Q22 | 3.69 | 1 |
| TPCH-Q22 | 3.58 | 1 |
| TPCH-Q22 | 3.24 | 1 |
+----------+----------------+-----------+
| TPCH-Q3 | 3.58 | 1 |
| TPCH-Q3 | 3.68 | 1 |
| TPCH-Q3 | 3.89 | 1 |
+----------+----------------+-----------+
| TPCH-Q4 | 1.37 | 1 |
| TPCH-Q4 | 1.37 | 1 |
| TPCH-Q4 | 1.32 | 1 |
+----------+----------------+-----------+
| TPCH-Q5 | 1.68 | 1 |
| TPCH-Q5 | 1.69 | 1 |
| TPCH-Q5 | 1.65 | 1 |
+----------+----------------+-----------+
| TPCH-Q6 | 0.46 | 1 |
| TPCH-Q6 | 0.47 | 1 |
| TPCH-Q6 | 0.47 | 1 |
+----------+----------------+-----------+
| TPCH-Q7 | 2.54 | 1 |
| TPCH-Q7 | 2.49 | 1 |
| TPCH-Q7 | 2.54 | 1 |
+----------+----------------+-----------+
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 1.90 | 1 |
+----------+----------------+-----------+
| TPCH-Q9 | 12.72 | 1 |
| TPCH-Q9 | 12.02 | 1 |
| TPCH-Q9 | 12.67 | 1 |
+----------+----------------+-----------+
Table Format: iceberg/lz4/block
+----------+----------------+-----------+
| Query | Time Taken (s) | Client ID |
+----------+----------------+-----------+
| TPCH-Q1 | 7.51 | 1 |
| TPCH-Q1 | 7.61 | 1 |
| TPCH-Q1 | 7.60 | 1 |
+----------+----------------+-----------+
| TPCH-Q10 | 3.79 | 1 |
| TPCH-Q10 | 3.73 | 1 |
| TPCH-Q10 | 3.67 | 1 |
+----------+----------------+-----------+
| TPCH-Q11 | 0.75 | 1 |
| TPCH-Q11 | 0.88 | 1 |
| TPCH-Q11 | 0.69 | 1 |
+----------+----------------+-----------+
| TPCH-Q12 | 1.18 | 1 |
| TPCH-Q12 | 1.13 | 1 |
| TPCH-Q12 | 1.04 | 1 |
+----------+----------------+-----------+
| TPCH-Q13 | 10.11 | 1 |
| TPCH-Q13 | 9.76 | 1 |
| TPCH-Q13 | 10.31 | 1 |
+----------+----------------+-----------+
| TPCH-Q14 | 1.64 | 1 |
| TPCH-Q14 | 1.64 | 1 |
| TPCH-Q14 | 1.53 | 1 |
+----------+----------------+-----------+
| TPCH-Q15 | 1.44 | 1 |
| TPCH-Q15 | 1.39 | 1 |
| TPCH-Q15 | 1.35 | 1 |
+----------+----------------+-----------+
| TPCH-Q16 | 2.24 | 1 |
| TPCH-Q16 | 2.15 | 1 |
| TPCH-Q16 | 2.15 | 1 |
+----------+----------------+-----------+
| TPCH-Q17 | 1.00 | 1 |
| TPCH-Q17 | 1.00 | 1 |
| TPCH-Q17 | 0.99 | 1 |
+----------+----------------+-----------+
| TPCH-Q18 | 15.14 | 1 |
| TPCH-Q18 | 14.94 | 1 |
| TPCH-Q18 | 15.04 | 1 |
+----------+----------------+-----------+
| TPCH-Q19 | 1.63 | 1 |
| TPCH-Q19 | 1.66 | 1 |
| TPCH-Q19 | 1.60 | 1 |
+----------+----------------+-----------+
| TPCH-Q2 | 0.69 | 1 |
| TPCH-Q2 | 0.75 | 1 |
| TPCH-Q2 | 0.69 | 1 |
+----------+----------------+-----------+
| TPCH-Q20 | 1.10 | 1 |
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.21 | 1 |
+----------+----------------+-----------+
| TPCH-Q21 | 25.38 | 1 |
| TPCH-Q21 | 25.48 | 1 |
| TPCH-Q21 | 25.18 | 1 |
+----------+----------------+-----------+
| TPCH-Q22 | 3.54 | 1 |
| TPCH-Q22 | 3.59 | 1 |
| TPCH-Q22 | 3.49 | 1 |
+----------+----------------+-----------+
| TPCH-Q3 | 3.95 | 1 |
| TPCH-Q3 | 3.96 | 1 |
| TPCH-Q3 | 3.79 | 1 |
+----------+----------------+-----------+
| TPCH-Q4 | 1.43 | 1 |
| TPCH-Q4 | 1.43 | 1 |
| TPCH-Q4 | 1.43 | 1 |
+----------+----------------+-----------+
| TPCH-Q5 | 1.84 | 1 |
| TPCH-Q5 | 1.80 | 1 |
| TPCH-Q5 | 1.71 | 1 |
+----------+----------------+-----------+
| TPCH-Q6 | 0.56 | 1 |
| TPCH-Q6 | 0.47 | 1 |
| TPCH-Q6 | 0.53 | 1 |
+----------+----------------+-----------+
| TPCH-Q7 | 2.75 | 1 |
| TPCH-Q7 | 2.59 | 1 |
| TPCH-Q7 | 2.49 | 1 |
+----------+----------------+-----------+
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 2.00 | 1 |
| TPCH-Q8 | 1.91 | 1 |
+----------+----------------+-----------+
| TPCH-Q9 | 13.34 | 1 |
| TPCH-Q9 | 12.18 | 1 |
| TPCH-Q9 | 12.71 | 1 |
+----------+----------------+-----------+
Change-Id: Ic8bbc5209e7e649f67a48144a2844b35d9f9c7f1
Reviewed-on: http://gerrit.cloudera.org:8080/19114
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> load-data.py can generate test datasets for TPC-H and TPC-DS stored as Iceberg
> ------------------------------------------------------------------------------
>
> Key: IMPALA-11607
> URL: https://issues.apache.org/jira/browse/IMPALA-11607
> Project: IMPALA
> Issue Type: Test
> Reporter: LiPenglin
> Assignee: LiPenglin
> Priority: Major
>
> At present, we use
> https://cwiki.apache.org/confluence/display/IMPALA/Impala+Test+Data to
> generate test data. Impala 4.1.0 has officially supported Iceberg tables, but
> we don't have tools to generate Iceberg test data for TPC-H and TPC-DS, so we
> should use similar commands to generate data, e.g: "/bin/load-data.py -w
> tpcds -f -s 100 --table_formats=text/none,iceberg/none".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
