[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 5: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 30 Jun 2020 13:06:32 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..

IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

This patch adds the following 12 TPCDS queries to the class of
TestTpcdsDecimalV2Query: Q26, Q30, Q31, Q47, Q48, Q57, Q58, Q59, Q63,
Q83, Q85, and Q89. All the queries except for Q31 are added to the class
of TestTpcdsQuery as well because Impala returns one fewer row than
expected for TestTpcdsQuery::test_tpcds_q31(), which requires further
investigation.

To verify whether or not the returned result set from Impala for a given
query is correct, we compare the result set with that produced by the
HiveServer2 (HS2) in Impala's mini-cluster. We could execute SQL
statements in HS2 via Beeline, HS2's command line shell, which could be
launched by the following command.

beeline -u "jdbc:hive2://localhost:11050/default"

We note that among these 12 queries, the execution of Q31, Q58, and Q83
result in the error of "Counters limit exceeded" by TEZ. To work around
this problem, for these 3 queries we have to execute the following
statement before running them to increase the default number of
counters, which is set to 120.

set tez.counters.max=1200

On the other hand, the table of 'reason' is referenced by Q85. This
table was not referenced by any TPCDS query before this patch and thus
was not created. In this regard, in this patch we also modify
tpcds_schema_template.sql to create this additional table along with its
data.

Testing:
- Verified that this patch passes the exhaustive tests in the DEBUG
  build.

Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Reviewed-on: http://gerrit.cloudera.org:8080/16119
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q26.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q30.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q31.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q47.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q48.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q57.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q58.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q59.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q63.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q83.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q85.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q89.test
A testdata/workloads/tpcds/queries/tpcds-q26.test
A testdata/workloads/tpcds/queries/tpcds-q30.test
A testdata/workloads/tpcds/queries/tpcds-q47.test
A testdata/workloads/tpcds/queries/tpcds-q48.test
A testdata/workloads/tpcds/queries/tpcds-q57.test
A testdata/workloads/tpcds/queries/tpcds-q58.test
A testdata/workloads/tpcds/queries/tpcds-q59.test
A testdata/workloads/tpcds/queries/tpcds-q63.test
A testdata/workloads/tpcds/queries/tpcds-q83.test
A testdata/workloads/tpcds/queries/tpcds-q85.test
A testdata/workloads/tpcds/queries/tpcds-q89.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
26 files changed, 2,622 insertions(+), 1 deletion(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-30 Thread Quanlong Huang (Code Review)
Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 30 Jun 2020 07:56:48 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6076/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 30 Jun 2020 07:57:39 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 30 Jun 2020 07:57:38 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-29 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6460/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 30 Jun 2020 05:34:16 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-29 Thread Fang-Yu Rao (Code Review)
Fang-Yu Rao has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..

IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

This patch adds the following 12 TPCDS queries to the class of
TestTpcdsDecimalV2Query: Q26, Q30, Q31, Q47, Q48, Q57, Q58, Q59, Q63,
Q83, Q85, and Q89. All the queries except for Q31 are added to the class
of TestTpcdsQuery as well because Impala returns one fewer row than
expected for TestTpcdsQuery::test_tpcds_q31(), which requires further
investigation.

To verify whether or not the returned result set from Impala for a given
query is correct, we compare the result set with that produced by the
HiveServer2 (HS2) in Impala's mini-cluster. We could execute SQL
statements in HS2 via Beeline, HS2's command line shell, which could be
launched by the following command.

beeline -u "jdbc:hive2://localhost:11050/default"

We note that among these 12 queries, the execution of Q31, Q58, and Q83
result in the error of "Counters limit exceeded" by TEZ. To work around
this problem, for these 3 queries we have to execute the following
statement before running them to increase the default number of
counters, which is set to 120.

set tez.counters.max=1200

On the other hand, the table of 'reason' is referenced by Q85. This
table was not referenced by any TPCDS query before this patch and thus
was not created. In this regard, in this patch we also modify
tpcds_schema_template.sql to create this additional table along with its
data.

Testing:
- Verified that this patch passes the exhaustive tests in the DEBUG
  build.

Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q26.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q30.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q31.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q47.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q48.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q57.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q58.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q59.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q63.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q83.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q85.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q89.test
A testdata/workloads/tpcds/queries/tpcds-q26.test
A testdata/workloads/tpcds/queries/tpcds-q30.test
A testdata/workloads/tpcds/queries/tpcds-q47.test
A testdata/workloads/tpcds/queries/tpcds-q48.test
A testdata/workloads/tpcds/queries/tpcds-q57.test
A testdata/workloads/tpcds/queries/tpcds-q58.test
A testdata/workloads/tpcds/queries/tpcds-q59.test
A testdata/workloads/tpcds/queries/tpcds-q63.test
A testdata/workloads/tpcds/queries/tpcds-q83.test
A testdata/workloads/tpcds/queries/tpcds-q85.test
A testdata/workloads/tpcds/queries/tpcds-q89.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
26 files changed, 2,622 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16119/4
--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-28 Thread Quanlong Huang (Code Review)
Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16119/2/testdata/datasets/tpcds/tpcds_schema_template.sql
File testdata/datasets/tpcds/tpcds_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/16119/2/testdata/datasets/tpcds/tpcds_schema_template.sql@1142
PS2, Line 1142: r_reason_id   CHAR(16)
  : r_reason_desc CHAR(100)
nit: Why not using STRING? I think at least STRING saves some space.



--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Mon, 29 Jun 2020 02:56:10 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-28 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16119 )

Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6445/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sun, 28 Jun 2020 23:34:06 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

2020-06-28 Thread Fang-Yu Rao (Code Review)
Fang-Yu Rao has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16119


Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test 
suite
..

IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite

This patch adds the following 12 TPCDS queries to the class of
TestTpcdsDecimalV2Query: Q26, Q30, Q31, Q47, Q48, Q57, Q58, Q59, Q63,
Q83, Q85, and Q89. All the queries except for Q31 are added to the class
of TestTpcdsQuery as well because Impala returns one fewer row than
expected for TestTpcdsQuery::test_tpcds_q31(), which requires further
investigation.

To verify whether or not the returned result set from Impala for a given
query is correct, we compare the result set with that produced by the
HiveServer2 (HS2) in Impala's mini-cluster. We could execute SQL
statements in HS2 via Beeline, HS2's command line shell, which could be
launched by the following command.

beeline -u "jdbc:hive2://localhost:11050/default"

We note that among these 12 queries, the execution of Q31, Q58, and Q83
result in the error of "Counters limit exceeded" by TEZ. To work around
this problem, for these 3 queries we have to execute the following
statement before running them to increase the default number of
counters, which is set to 120.

set tez.counters.max=1200

On the other hand, the table of 'reason' is referenced by Q85. This
table was not referenced by any TPCDS query before this patch and thus
was not created. In this regard, in this patch we also modify
tpcds_schema_template.sql to create this additional table along with its
data.

Testing:
- Verified that this patch passes the exhaustive tests in the DEBUG
  build.

Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q26.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q30.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q31.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q47.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q48.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q57.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q58.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q59.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q63.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q83.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q85.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q89.test
A testdata/workloads/tpcds/queries/tpcds-q26.test
A testdata/workloads/tpcds/queries/tpcds-q30.test
A testdata/workloads/tpcds/queries/tpcds-q47.test
A testdata/workloads/tpcds/queries/tpcds-q48.test
A testdata/workloads/tpcds/queries/tpcds-q57.test
A testdata/workloads/tpcds/queries/tpcds-q58.test
A testdata/workloads/tpcds/queries/tpcds-q59.test
A testdata/workloads/tpcds/queries/tpcds-q63.test
A testdata/workloads/tpcds/queries/tpcds-q83.test
A testdata/workloads/tpcds/queries/tpcds-q85.test
A testdata/workloads/tpcds/queries/tpcds-q89.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
26 files changed, 2,622 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16119/2
--
To view, visit http://gerrit.cloudera.org:8080/16119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf
Gerrit-Change-Number: 16119
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong