Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20756 )
Change subject: IMPALA-12601: Add a fully partitioned TPC-DS database ...................................................................... Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/20756/2/testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql File testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/20756/2/testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql@483 PS2, Line 483: -- Unset the following MAX_SCAN_RANGE_LENGTH if loading from high scale TPC-DS base db. : SET MAX_SCAN_RANGE_LENGTH=1mb; : SET SORT_RUN_BYTES_LIMIT=512mb; : SET MT_DOP=4; What do these settings do? Do we still need them? http://gerrit.cloudera.org:8080/#/c/20756/2/testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql@715 PS2, Line 715: ---- DEPENDENT_LOAD : INSERT OVERWRITE TABLE {db_name}{db_suffix}.{table_name} PARTITION (ss_sold_date_sk) : SELECT ss_sold_time_sk, We don't specify the "partitioned_insert:" multi-statement load here. Is that because this only does parquet (and thus uses Impala)? I'm a bit unclear whether we need the multi-statement anymore for regular TPC-DS. Sahil did an optimization for Hive that helped: https://github.com/apache/impala/commit/00ef25888080bb1ec792c01177ab6ebcff447c5d -- To view, visit http://gerrit.cloudera.org:8080/20756 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a2e66c405639554f325ae78c66628d464f6c453 Gerrit-Change-Number: 20756 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Wed, 06 Dec 2023 17:41:54 +0000 Gerrit-HasComments: Yes
