[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Reviewed-on: http://gerrit.cloudera.org:8080/21153 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py A tests/custom_cluster/test_sys_db.py M tests/util/workload_management.py 5 files changed, 1,086 insertions(+), 490 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 10 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 27 Mar 2024 04:46:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 23:34:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10434/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 23:34:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 23:25:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15683/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 22:53:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 8: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@51 PS6, Line 51: > There are a few fields in the completed queries table that are different ba Oh right! Might be worth a comment for the next time I forget. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 22:39:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#8). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py A tests/custom_cluster/test_sys_db.py M tests/util/workload_management.py 5 files changed, 1,086 insertions(+), 490 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/8 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 8: (5 comments) http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@51 PS6, Line 51: > I wouldn't expect any difference in behavior between Beeswax and HS2, from There are a few fields in the completed queries table that are different based on the client protocol. The TestQueryLogTableAll class is the only test class that runs using both the beeswax and hs2 dimensions. This class contains unique tests that I would like to test on both client protocols to cover any corner cases. For example, there is only 1 dml query and 1 ddl that are asserted. I test on both protocols for those. I also test one invalid query and all combinations of ignore queries to make sure that changing the client protocol doesn't actually run on some unexpected code path that should not be executed. I was very intentional to only do duplication where I thought it added unique test coverage. http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@97 PS6, Line 97: # causing the execution to take a very long time. > Could you comment that we use async to avoid fetching the large result stri Done http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@383 PS6, Line 383: cluster_size=3, > This isn't specific to query_log, could we move the sys db tests to a new f Done http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py@208 PS7, Line 208: res = client.execute(select_sql) > Why do you sleep between runs? The original issue is that running queries too quickly did not cause the cache to be populated. I modified this code to check the complete-queries.written metric after each query. That will ensure there is some down time between queries. http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py@214 PS7, Line 214: self.cluster.get_first_impalad().service.wait_for_metric_value( > What are you actually waiting on. Is there a data-cache metric we can watch Time is needed for the cache to be populated or else it is not used. I modified this code to check a metric instead of sleeping. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 22:28:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/21153 )
Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py
File tests/custom_cluster/test_query_log.py:
http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@51
PS6, Line 51: cls.PROTOCOL_BEESWAX, cls.PROTOCOL_HS2))
I wouldn't expect any difference in behavior between Beeswax and HS2, from the
frontend these are very standard queries. Was there a specific reason to do
both?
http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@97
PS6, Line 97: handle = client.execute_async("select
'{0}'".format(rand_long_str))
Could you comment that we use async to avoid fetching the large result string.
http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@383
PS6, Line 383: def test_query_log_table_create_table_sys_db_blocked(self,
vector):
This isn't specific to query_log, could we move the sys db tests to a new file?
http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py
File tests/custom_cluster/test_query_log.py:
http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py@208
PS7, Line 208: sleep(1)
Why do you sleep between runs?
http://gerrit.cloudera.org:8080/#/c/21153/7/tests/custom_cluster/test_query_log.py@214
PS7, Line 214: sleep(10)
What are you actually waiting on. Is there a data-cache metric we can watch for
this?
--
To view, visit http://gerrit.cloudera.org:8080/21153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b
Gerrit-Change-Number: 21153
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Tue, 26 Mar 2024 16:58:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15677/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 15:55:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@881 PS6, Line 881: TestQueryLogTableBas > This should be TestQueryLogTableBase. Otherwise, it inherits and runs all t Done -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 26 Mar 2024 15:32:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#7). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py M tests/util/workload_management.py 4 files changed, 1,053 insertions(+), 491 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/7 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/6/tests/custom_cluster/test_query_log.py@881 PS6, Line 881: TestQueryLogTableAll This should be TestQueryLogTableBase. Otherwise, it inherits and runs all test methods of TestQueryLogTableAll. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 6 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 25 Mar 2024 18:42:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15636/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 6 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 19:54:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#6). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py M tests/util/workload_management.py 4 files changed, 1,053 insertions(+), 491 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/6 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 6 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15621/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 20:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15620/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 20:49:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@635 PS1, Line 635: catalogd_args="--enable_workload_m > https://impala.apache.org/docs/build/html/topics/impala_spool_query_results Ack http://gerrit.cloudera.org:8080/#/c/21153/3/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/3/tests/custom_cluster/test_query_log.py@822 PS3, Line 822: TestQueryLogTableAll I think this should extend TestQueryLogTableBase? -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 20:28:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@635 PS1, Line 635: catalogd_args="--enable_workload_m > https://impala.apache.org/docs/build/html/topics/impala_spool_query_results Based on the comments in IMPALA-12932 that SPOOL_QUERY_RESULTS should be true and the docs need changing, I am deleting this line. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 20:27:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#5). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py 3 files changed, 982 insertions(+), 485 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/5 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#4). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py 3 files changed, 983 insertions(+), 485 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/4 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15613/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 19:06:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#3). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py 3 files changed, 983 insertions(+), 485 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/3 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21153/3/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/21153/3/tests/custom_cluster/test_query_log.py@390 PS3, Line 390: r flake8: F841 local variable 'res' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/21153/3/tests/custom_cluster/test_query_log.py@405 PS3, Line 405: r flake8: F841 local variable 'res' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 21 Mar 2024 18:44:23 + Gerrit-HasComments: Yes
