[Impala-ASF-CR] IMPALA-12705: Add /catalog ha info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog_ha_info page on Statestore to show catalog HA information .. Patch Set 14: (4 comments) http://gerrit.cloudera.org:8080/#/c/21418/14/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/21418/14/tests/webserver/test_web_pages.py@1141 PS14, Line 1141: ImpalaTestSuite This is not CustomClusterTestSuite. The Impala cluster is not restarted for each test case. http://gerrit.cloudera.org:8080/#/c/21418/14/tests/webserver/test_web_pages.py@1171 PS14, Line 1171: disable_catalog_ha function name should be started with test_ http://gerrit.cloudera.org:8080/#/c/21418/14/tests/webserver/test_web_pages.py@1174 PS14, Line 1174: @CustomClusterTestSuite.with_args(start_args="--enable_catalogd_ha") Could we set parameters if the class is not CustomClusterTestSuite? Is Impala cluster restarted with given parameter? http://gerrit.cloudera.org:8080/#/c/21418/14/tests/webserver/test_web_pages.py@1175 PS14, Line 1175: enable_catalog_ha function name should be started with test_ -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 14 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Fri, 31 May 2024 06:28:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13106: Support larger imported query profile sizes through compression
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21463 ) Change subject: IMPALA-13106: Support larger imported query profile sizes through compression .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21463/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21463/2//COMMIT_MSG@16 PS2, Line 16: Should we make 'compression' as option to be enabled? -- To view, visit http://gerrit.cloudera.org:8080/21463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4f31beb9cac89051460bf764b6d50c3933bd03 Gerrit-Change-Number: 21463 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 30 May 2024 19:06:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12705: Add /catalog ha info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog_ha_info page on Statestore to show catalog HA information .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/21418/13/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/21418/13/tests/webserver/test_web_pages.py@1191 PS13, Line 1191: else: Is this condition triggered during test? -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 13 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Thu, 30 May 2024 18:49:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12705: Add /catalog ha info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog_ha_info page on Statestore to show catalog HA information .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/21418/13/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/21418/13/tests/webserver/test_web_pages.py@1175 PS13, Line 1175: flag nit: ha_flags -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 13 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Thu, 30 May 2024 18:23:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, this patch checks number of instances in received TExecPlanFragment on executor. The query will not be started if number of instances equals 0. Also adds DCHECK on coordinator side to make sure it does not send TExecPlanFragment without any instance. Testing: - Passed core tests. - Passed exhaustive tests in debug build. The new DCHECKs were not hit. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/runtime/coordinator-backend-state.cc M be/src/service/control-service.cc 2 files changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/6 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. Patch Set 6: Code-Review+2 carry +2 from Riza -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 29 May 2024 17:56:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21458/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21458/5//COMMIT_MSG@21 PS5, Line 21: hit. > nit: "hit" is the right past tense verb. fixed -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 29 May 2024 17:55:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, this patch checks number of instances in received TExecPlanFragment on executor. The query will not be started if number of instances equals 0. Also adds DCHECK on coordinator side to make sure it does not send TExecPlanFragment without any instance. Testing: - Passed core tests. - Passed exhaustive tests in debug build. The new DCHECKs were not hitted. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/runtime/coordinator-backend-state.cc M be/src/service/control-service.cc 2 files changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/5 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21458/3/be/src/service/control-service.cc File be/src/service/control-service.cc: http://gerrit.cloudera.org:8080/#/c/21458/3/be/src/service/control-service.cc@154 PS3, Line 154: UNLIKELY(fragment_info.fragments.size() == 0 : || fragment_info.fragment_instance_ctxs.size() == 0 > nit: wrap this with UNLIKELY() Fixed -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 28 May 2024 23:11:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, this patch checks number of instances in received TExecPlanFragment on executor. The query will not be started if number of instances equals 0. Also adds DCHECK on coordinator side to make sure it does not send TExecPlanFragment without any instance. Testing: - Passed core tests. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/runtime/coordinator-backend-state.cc M be/src/service/control-service.cc 2 files changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/4 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, this patch checks number of instances in received TExecPlanFragment on executor. The query will not be started if number of instances equals 0. Also adds DCHECK on coordinator side to make sure it does not send TExecPlanFragment without any instance. Testing: - Passed core tests. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/runtime/coordinator-backend-state.cc M be/src/service/control-service.cc 2 files changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/3 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, this patch checks number of instances in received TExecPlanFragment on executor. The query will not be started if number of instances equals 0. Also adds DCHECK on coordinator side to make sure it does not send TExecPlanFragment without any instance. Testing: - Passed core tests. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/runtime/coordinator-backend-state.cc M be/src/service/control-service.cc 2 files changed, 12 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/2 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21458 ) Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21458/1/be/src/service/control-service.cc File be/src/service/control-service.cc: http://gerrit.cloudera.org:8080/#/c/21458/1/be/src/service/control-service.cc@154 PS1, Line 154: fragment_info.fragments.size() == 0 : || fragment_info.fragment_instance_ctxs.size() == 0 > Can you also add DCHECK in Coordinator::BackendState::SetRpcParams verifyin Added DCHECK in Coordinator::BackendState::SetRpcParams as suggested. -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 28 May 2024 20:45:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information .. Patch Set 12: overall looks good. Could you add unit-test for new web page? -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 12 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 28 May 2024 04:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information .. Patch Set 10: (4 comments) http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore-catalogd-mgr.h File be/src/statestore/statestore-catalogd-mgr.h: http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore-catalogd-mgr.h@87 PS10, Line 87: const remove const http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore.h File be/src/statestore/statestore.h: http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore.h@514 PS10, Line 514: const remove const since subscriber_type_ is not const variable. http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore.h@517 PS10, Line 517: const remove const since catalogd_version is not const variable http://gerrit.cloudera.org:8080/#/c/21418/10/be/src/statestore/statestore.h@519 PS10, Line 519: const remove const -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 27 May 2024 17:32:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog-ha-info page on Statestore to show catalog HA information .. Patch Set 10: The build failures were caused by following warning: /home/ubuntu/Impala/be/src/statestore/statestore-catalogd-mgr.h:87:3: warning: 'const' type qualifier on return type has no effect [clang-diagnostic-ignored-qualifiers] /home/ubuntu/Impala/be/src/statestore/statestore.h:514:5: warning: 'const' type qualifier on return type has no effect [clang-diagnostic-ignored-qualifiers] /home/ubuntu/Impala/be/src/statestore/statestore.h:517:5: warning: 'const' type qualifier on return type has no effect [clang-diagnostic-ignored-qualifiers] /home/ubuntu/Impala/be/src/statestore/statestore.h:519:5: warning: 'const' type qualifier on return type has no effect [clang-diagnostic-ignored-qualifiers] -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 27 May 2024 17:25:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13107: Don't start query on executor if instance number equals 0
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21458 Change subject: IMPALA-13107: Don't start query on executor if instance number equals 0 .. IMPALA-13107: Don't start query on executor if instance number equals 0 In bad networking condition, TExecPlanFragmentInfo in KRPC messages received by executors could be truncated due to KRPC failures, but truncation may not cause thrift deserialization error. The invalid TExecPlanFragmentInfo causes Impala daemon to crash. To avoid crash, query with number of instances as 0 should not be started on executor. Testing: - Passed core tests. Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 --- M be/src/service/control-service.cc 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/21458/1 -- To view, visit http://gerrit.cloudera.org:8080/21458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie92ee120f1e9369f8dc2512792a05b7f8be5f007 Gerrit-Change-Number: 21458 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13105: Fix multiple imported query profiles fail to import/clear at once
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21450 ) Change subject: IMPALA-13105: Fix multiple imported query profiles fail to import/clear at once .. Patch Set 1: Code-Review+1 Ran on my local machine. It's working as expected. -- To view, visit http://gerrit.cloudera.org:8080/21450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42470fecd0cff6e193f080102575e51d86a2d562 Gerrit-Change-Number: 21450 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 22 May 2024 17:38:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12705: Add /catalog-ha-info page on Starstore to show catalog HA information
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21418 ) Change subject: IMPALA-12705: Add /catalog-ha-info page on Starstore to show catalog HA information .. Patch Set 6: (7 comments) Thanks to add this enhancement for catalog HA. http://gerrit.cloudera.org:8080/#/c/21418/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21418/6//COMMIT_MSG@7 PS6, Line 7: Starstore nit: typo statestore http://gerrit.cloudera.org:8080/#/c/21418/6//COMMIT_MSG@9 PS6, Line 9: Starstor typo statestore http://gerrit.cloudera.org:8080/#/c/21418/6//COMMIT_MSG@11 PS6, Line 11: Subscribers nit: wrap up long line http://gerrit.cloudera.org:8080/#/c/21418/6//COMMIT_MSG@20 PS6, Line 20: Should we add unit-test for this new page? http://gerrit.cloudera.org:8080/#/c/21418/6/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/21418/6/be/src/statestore/statestore.cc@652 PS6, Line 652: catalogd_version_ = catalogd_version; need to acquire lock before updating. http://gerrit.cloudera.org:8080/#/c/21418/6/be/src/statestore/statestore.cc@2245 PS6, Line 2245: subscriber.second->subscriber_type() == TStatestoreSubscriberType::CATALOGD : || subscriber.second->IsCoordinator() can replace with subscriber.second->IsSubscribedCatalogdChange() http://gerrit.cloudera.org:8080/#/c/21418/6/common/thrift/StatestoreService.thrift File common/thrift/StatestoreService.thrift: http://gerrit.cloudera.org:8080/#/c/21418/6/common/thrift/StatestoreService.thrift@222 PS6, Line 222: registration_time_ms This is timestamp, not time period. Name with appendix '_ms' sounds like period, i.e. 'time in milliseconds'. -- To view, visit http://gerrit.cloudera.org:8080/21418 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If85f6a827ae8180d13caac588b92af0511ac35e3 Gerrit-Change-Number: 21418 Gerrit-PatchSet: 6 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 21 May 2024 01:14:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13809: Revert "IMPALA-12559: Support x5c Parameter for RSA JSON"
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21434 ) Change subject: IMPALA-13809: Revert "IMPALA-12559: Support x5c Parameter for RSA JSON" .. Patch Set 1: Patch https://gerrit.cloudera.org/#/c/21432/ fixed this build issue. It was just merged so we don't need to revert "IMPALA-12559" -- To view, visit http://gerrit.cloudera.org:8080/21434 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaedfe72429eef61fd5bc18cb2939f468ee3da1ec Gerrit-Change-Number: 21434 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 May 2024 13:37:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559 (part 2): Fix build issue for different versions of openssl
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21432 ) Change subject: IMPALA-12559 (part 2): Fix build issue for different versions of openssl .. IMPALA-12559 (part 2): Fix build issue for different versions of openssl Previous patch calls OpenSSL API X509_get0_tbs_sigalg() which is not available in the version of OpenSSL in ToolChain. It causes build failures. This patch fixes the issue by calling X509_get_signature_nid(). Testing: - Passed jwt-test unit-test and end-end unit-test. Change-Id: I62b9f0c00f91c2b13be30c415e3f1ebd0e1bd2bc Reviewed-on: http://gerrit.cloudera.org:8080/21432 Reviewed-by: gaurav singh Tested-by: Impala Public Jenkins Reviewed-by: Abhishek Rawat --- M be/src/util/jwt-util.cc 1 file changed, 8 insertions(+), 2 deletions(-) Approvals: gaurav singh: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Abhishek Rawat: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I62b9f0c00f91c2b13be30c415e3f1ebd0e1bd2bc Gerrit-Change-Number: 21432 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559 (part 2): Fix build issue for different versions of openssl
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21432 Change subject: IMPALA-12559 (part 2): Fix build issue for different versions of openssl .. IMPALA-12559 (part 2): Fix build issue for different versions of openssl Previous patch calls OpenSSL API X509_get0_tbs_sigalg() which is not available in the version of OpenSSL in ToolChain. It causes build failures. This patch fixes the issue by calling X509_get_signature_nid(). Testing: - Passed jwt-test unit-test and end-end unit-test. Change-Id: I62b9f0c00f91c2b13be30c415e3f1ebd0e1bd2bc --- M be/src/util/jwt-util.cc 1 file changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/21432/1 -- To view, visit http://gerrit.cloudera.org:8080/21432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I62b9f0c00f91c2b13be30c415e3f1ebd0e1bd2bc Gerrit-Change-Number: 21432 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 27: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 27 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 23:54:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 27: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 27 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 23:48:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 26: (2 comments) http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@161 PS26, Line 161: IfExists Don't need to make this change. Instead, set createJWKSForWebServer_ as false to keep consistent with other test cases. http://gerrit.cloudera.org:8080/#/c/21382/26/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@566 PS26, Line 566: true Set as false -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 26 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 23:37:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 25: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 25 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 18:52:14 + Gerrit-HasComments: No
[Impala-ASF-CR] CDPD-68003: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: CDPD-68003: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 24: (1 comment) http://gerrit.cloudera.org:8080/#/c/21382/24//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21382/24//COMMIT_MSG@7 PS24, Line 7: CDPD-68003: Don't put downstream Jira here for upstream patch -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 24 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 18:16:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 23: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 23 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 17:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 22: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21382/22/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/21382/22/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@567 PS22, Line 567: String statestoreWebserverArgs = "--webserver_port=25010"; nit: it's unused -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 22 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 14 May 2024 16:42:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 21:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java:
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@573
PS21, Line 573: "", statestoreWebserverArgs, 0
If read jwks from local file, then don't need these three parameters, and don't
need variable statestoreWebserverArgs
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@585
PS21, Line 585: impala
Should be 'jwt-cpp.example.localhost'
http://gerrit.cloudera.org:8080/#/c/21382/21/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@598
PS21, Line 598: // Authenticate fails due JWT verification failure since the
RS256 public key cannot
: // be found in the JWKS.
: transport.setCustomHeaders(headers);
: try {
: openResp = client.OpenSession(openReq);
: } catch (Exception e) {
: verifyJwtAuthMetrics(3, 1);
: assertEquals(e.getMessage(), "HTTP Response code: 401");
: }
This negative test is dependent on the JWKS change in running time. It should
be removed since you read JWKS from local file now, and jwks will not be
changed in running time.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 21
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Tue, 14 May 2024 00:44:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 20: (1 comment) http://gerrit.cloudera.org:8080/#/c/21382/20/testdata/jwt/jwks_x5c_rs256.json File testdata/jwt/jwks_x5c_rs256.json: http://gerrit.cloudera.org:8080/#/c/21382/20/testdata/jwt/jwks_x5c_rs256.json@10 PS20, Line 10: "MIIE2jCCAsICAQ > I'm not seeing where the double quote is missing. The end of the line has The double quote is not shown in the UI of gerrit. I just downloaded the patch and checked this json file on local machine. There is double quote in the end of line. False alarm. -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 20 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 13 May 2024 16:22:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 20:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@158
PS20, Line 158: v = values[0];
enclose this line with { } to follow coding style.
http://gerrit.cloudera.org:8080/#/c/21382/20/be/src/util/jwt-util.cc@258
PS20, Line 258: GetString()
Line 251 allow different json type, here assume it's string. Please keep
consistent.
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java:
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@110
PS20, Line 110: JWKS_FILE_NAME
This function currently use fixed json file. You may add new input parameter
when using jwks_x5c_rs256.json
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@575
PS20, Line 575: impaladJwtArgs, "", statestoreWebserverArgs, 0)
add a new parameter to pass jwks_x5c_rs256.json
http://gerrit.cloudera.org:8080/#/c/21382/20/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@600
PS20, Line 600: // Update JWKS in the root directory of Web server.
: createTempJWKSInWebServerRootDir("jwks_x5c_rs256.json");
: // Sleep long enough for coordinator to update JWKS from
Web server.
: Thread.sleep(3000);
: // Authenticate fails due JWT verification failure since
the RS256 public key cannot
: // be found in the JWKS.
: transport.setCustomHeaders(headers);
: try {
: openResp = client.OpenSession(openReq);
: } catch (Exception e) {
: verifyJwtAuthMetrics(3, 1);
: assertEquals(e.getMessage(), "HTTP Response code: 401");
: }
This function block seems copying from testJwtAuthWithJwksHttpUrl(). In that
function, we add a negative test case by changing json file with unexpected
contents. Why do you callcreateTempJWKSInWebServerRootDir()
with"jwks_x5c_rs256.json"?
http://gerrit.cloudera.org:8080/#/c/21382/20/testdata/jwt/jwks_x5c_rs256.json
File testdata/jwt/jwks_x5c_rs256.json:
http://gerrit.cloudera.org:8080/#/c/21382/20/testdata/jwt/jwks_x5c_rs256.json@10
PS20, Line 10: "MIIE2jCCAsICAQ
missing double quote in the end of line
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 20
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Sat, 11 May 2024 00:46:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 5: It seems browser caching issue. It's working when using Chrome from another machine, or using Firefox from same machine. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 10 May 2024 16:39:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 10 May 2024 16:34:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21409 ) Change subject: IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables .. Patch Set 4: Thanks Riza and Abhishek. -- To view, visit http://gerrit.cloudera.org:8080/21409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572 Gerrit-Change-Number: 21409 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 10 May 2024 02:05:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Block push down of conjuncts with implicit
casting on base columns for jdbc tables
..
IMPALA-13018: Block push down of conjuncts with implicit casting on base
columns for jdbc tables
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by blocking such conjuncts with
implicit unsafe casting or casting to date/timestamp to be added into
offered predicate list for JDBC table.
Note that explicit casting on base columns are not allowed to
pushdown.
Testing:
- Add new planner unit-tests, including explicit casting, implicit
casting to date/timestamp, built-in functions, arithmetic
expressions.
The predicates which are accepted for JDBC are shown in plan under
"data source predicates" of DataSourceScanNode, predicates which
are not accepted for JDBC are shown in plan under "predicates" of
DataSourceScanNodes.
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
---
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M
testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M
testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test
M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test
M tests/query_test/test_tpcds_queries.py
5 files changed, 287 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21409/4
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Block push down of conjuncts with implicit
casting on base columns for jdbc tables
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21409/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21409/2//COMMIT_MSG@21
PS2, Line 21: Note that explicit casting on base columns are not allowed to
> nit:
Fixed
http://gerrit.cloudera.org:8080/#/c/21409/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java:
http://gerrit.cloudera.org:8080/#/c/21409/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@294
PS2, Line 294: // Other binary predicate scenarios are not accepted,
including:
> spelling:
Updated comments and added planner tests for explicit CAST, built-in functions,
arithmetic expressions. Those predicates, built-in function, expressions are
not pushed down to JDBC, e.g. not in "data source predicates", but in
"predicates" of DataSourceScaNode so that those casting, built-in functions and
expressions will be executed on coordinator.
http://gerrit.cloudera.org:8080/#/c/21409/2/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@301
PS2, Line 301: lExpr instanceof CastExpr) {
: CastExpr castExpr = (CastExpr)colExpr;
> nit: Any new planner test for this?
Added planner tests for explicit CAST, implicit casting to date/timestamp,
which shows those predicates are not pushed down to JDBC, e.g. not in "data
source predicates"
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 10 May 2024 00:56:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Block push down of conjuncts with implicit
casting on base columns for jdbc tables
..
IMPALA-13018: Block push down of conjuncts with implicit casting on base
columns for jdbc tables
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by blocking such conjuncts with
implicit unsafe casting or casting to date/timestamp to be added into
offered predicate list for JDBC table.
Note that explicit casting on base columns are not allowed to
pushdown.
Testing:
- Add new planner unit-tests, including explicit casting, implicit
casting to date/timestamp, built-in functions, arithmetic
expressions.
The predicates which are accepted for JDBC are shown in plan under
"data source predicates" of DataSourceScanNode, predicates which
are not accepted for JDBC are shown in plan under "predicates" of
DataSourceScanNodes.
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
---
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M
testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M
testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test
M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test
M tests/query_test/test_tpcds_queries.py
5 files changed, 264 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21409/3
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 5: Did a clean build and clean the browser cache. The issue was still there. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 09 May 2024 21:00:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 5: Yes, getQueryID() function is there. It was working on my local machine before patch set #4. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 09 May 2024 20:11:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 5: Tried patch set #5. In timeline tab, got following error in browser console: query_timeline?query_id=dd4d881b6dee0f20:bbe3296d=true:484 Uncaught ReferenceError: getQueryID is not defined at dbOpenReq.onsuccess (query_timeline?query_id=dd4d881b6dee0f20:bbe3296d=true:484:18) In Query table, got following error on console. Uncaught ReferenceError: getQueryID is not defined at dbOpenReq.onsuccess (query_stmt?query_id=dd4d881b6dee0f20:bbe3296d=true:381:18) -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 09 May 2024 20:04:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Block push down of conjuncts with implicit
casting on base columns for jdbc tables
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-13018: Block push down of conjuncts with implicit casting
on base columns for jdbc tables
> Maybe update the title to something generic, since this applies to other qu
Done.
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG@17
PS1, Line 17: But casting to Date/Timestamp for a column cannot be pushed down
to JDBC
> I'm wondering if this applies to other non Date/Timestamp types also? Can w
All explicate castings on base columns are not allowed in current
implementation so CAST (stringCol AS INT) >= 1234 will not be pushed down. For
Between, planner generate implicate casting between compatible data types.
Since string and integer are not compatible types, exception will be thrown if
comparing string with integer, like stringCol >= 1234 or stringCol >=
cast("1234" as INT).
We should allow implicate casting between compatible data types other than
date/timestamp, like numeric data types. For safety, unsafe casting will be
blocked.
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 09 May 2024 19:13:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13018: Block push down of conjuncts with implicit casting on base columns for jdbc tables
Wenzhe Zhou has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Block push down of conjuncts with implicit
casting on base columns for jdbc tables
..
IMPALA-13018: Block push down of conjuncts with implicit casting on base
columns for jdbc tables
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by blocking such conjuncts with
implicit unsafe casting or casting to date/timestamp to be added into
offered predicate list for JDBC table.
Note that explicate castings on base columns are not allowed to
pushdown.
Testing:
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
---
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M tests/query_test/test_tpcds_queries.py
2 files changed, 21 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21409/2
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 4: Tried the patch set 4. Timeline tab shows nothing for an imported json profile. Other three tabs work fine. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 17:37:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@22 PS6, Line 22: > Could you add test case in fe/test/java/org/apache/impala/customcluster/Jwt Please add end-end test case. http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc File be/src/util/jwt-util.cc: http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc@403 PS10, Line 403: nit: extra spaces -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 08 May 2024 16:18:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 3 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 07:13:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:39:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc@320
PS7, Line 320:
: Status status;
: JWTPublicKey* jwt_pub_key = nullptr;
: try {
: if (algorithm == "hs256") {
: jwt_pub_key = new HS256JWTPublicKey(algorithm,
it_k->second);
: } else if (algorithm == "hs
What's the reason to remove x5c?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@356
PS8, Line 356: if (it_n == kv_map.end() || it_e == kv_map.end()) {
: return Status("'n' and 'e' properties are required");
: } else if (it_n->second.empty() || it_e->second.empty()) {
: return Status("'n' and 'e' properties must be a non-empty
string");
: }
Are "n" and "e" optional when 'x5c' is provided?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@381
PS8, Line 381: return Status(Substitute("Invalid x5c certificate"));
add { } around return statement.
Should we free bio before return?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@407
PS8, Line 407:
nit: extra indent spaces.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Wed, 08 May 2024 00:12:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:49:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 2: Code-Review+1 (4 comments) Thanks to fix this issue. http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@19 PS2, Line 19: cause nit: causes http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@21 PS2, Line 21: fix nit: fixes http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@22 PS2, Line 22: is nit: are http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@25 PS2, Line 25: is nit: are -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:34:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl File www/query_profile.tmpl: http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl@53 PS2, Line 53: nit: indent with two spaces for this function block to keep consistent. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 21:22:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 1: | Maybe a better solution is to Start query_events_ at start_time_us() It's better to keep consistent. Could we run exhaustive test on ARM to verify the fixing? -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 20:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21409 ) Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@300 PS1, Line 300: castExpr.getType().isDateOrTimeType() > nit: Just want to double check, is CastExpr OK for types other than Date/Ti Other numeric and string types are ok. For future release, we need to add TExpr to TBinaryPredicate. -- To view, visit http://gerrit.cloudera.org:8080/21409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572 Gerrit-Change-Number: 21409 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 19:31:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Wenzhe Zhou has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21409
Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table
..
IMPALA-13018: Fix test_tpcds_q80a for JDBC table
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by not adding such conjuncts to
offered predicate list for JDBC table.
Testing:
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
---
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M tests/query_test/test_tpcds_queries.py
2 files changed, 18 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21409/1
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG@14 PS1, Line 14: With the current patch "Query Profile" tab will also be supported. > Tried to run test with your patch on my local machine. But did not see the I messed up my local environment. It did show the "Query Profile" tab after clean build. Tried to switch tabs for an imported query. When switching from active tab "Query Profile" to other three tabs, it shows nothing with query-id as :. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 00:30:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG@14 PS1, Line 14: With the current patch "Query Profile" tab will also be supported. Tried to run test with your patch on my local machine. But did not see the new tab "Query Profile", still saw three tabs for imported queries. http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG@21 PS1, Line 21: converts nit: wrap around long line. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 00:02:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13053: Update test to use ORC files
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21391 ) Change subject: IMPALA-13053: Update test to use ORC files .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f5bdbb86af0e651d189217a18882d5eda1098d5 Gerrit-Change-Number: 21391 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 03 May 2024 17:54:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys .. Patch Set 7: (4 comments) http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc File be/src/util/jwt-util.cc: http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@217 PS6, Line 217: Value& json_value = j unused parameter. From function name, this function only works for array property http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@256 PS6, Line 256: value[i] = json_value[i].GetString(); \ : } \ : > Done The for loop block should be indented with 4 spaces. http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@323 PS6, Line 323: if (it_x5c != kv_map.end()) > We check each key regardless to validate them. If "k" is invalid, then the Is it possible there is no 'k' property when 'x5c' is provided? http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc File be/src/util/jwt-util.cc: http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc@55 PS7, Line 55: array_type nit: use upper case for constants -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 02 May 2024 18:20:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21350 ) Change subject: IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fb357016d72a363ee5016f7881b0f6b0426aff5 Gerrit-Change-Number: 21350 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 01 May 2024 23:05:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter in JSON Web Keys
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter in JSON Web Keys
..
Patch Set 6:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@12
PS6, Line 12: supports a single x5c certificate
What's the reason to support single x5c certificate?
http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@22
PS6, Line 22: verify jwt with x5c certificate.
Could you add test case in
fe/test/java/org/apache/impala/customcluster/JwtHttpTest.java?
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@51
PS6, Line 51: x5c certificate
Is there hard limit for number of x5s certificate in RFC?
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@150
PS6, Line 150: "Array"
nit: define a constant
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@200
PS6, Line 200: is_array
this variable is unused
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@209
PS6, Line 209:
nit: extra indent spaces
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@226
PS6, Line 226:
nit: extra indent spaces
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@245
PS6, Line 245: A true return value
function return Status, not boolean value
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@256
PS6, Line 256: for (size_t i = 0; i < json_value.Size() && i <
MAX_X5C_CERTIFICATES; i++) { \
: value[i] = json_value[i].GetString();
\
: }
nit: indent
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@323
PS6, Line 323: auto it_x5c = kv_map.find("x5c");
since 'x5c' has priority over 'k', could we check 'x5c' before checking 'k'?
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@385
PS6, Line 385: auto it_x5c = kv_map.find("x5c");
: if (it_x5c != kv_map.end())
: pub_key_str =
jwt::helper::convert_base64_der_to_pem(it_x5c->second);
: else
: pub_key_str = pub_key;
since 'x5c' has priority over 'n' and 'e', could we check 'x5c' first?
same comments for other types of KeyBuilder.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 6
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 01 May 2024 22:36:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 18: (8 comments) http://gerrit.cloudera.org:8080/#/c/21304/17//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21304/17//COMMIT_MSG@11 PS17, Line 11: Note > nit: Note that ... fixed http://gerrit.cloudera.org:8080/#/c/21304/17//COMMIT_MSG@19 PS17, Line 19: 5 minutes > Should this duration be configurable? Yes, added a flag variable. http://gerrit.cloudera.org:8080/#/c/21304/17//COMMIT_MSG@26 PS17, Line 26: > nit: Note that ... fixed http://gerrit.cloudera.org:8080/#/c/21304/17/be/src/service/frontend.cc File be/src/service/frontend.cc: http://gerrit.cloudera.org:8080/#/c/21304/17/be/src/service/frontend.cc@99 PS17, Line 99: DEFINE_int32(dbcp_max_conn_pool_size, 8, > Is this a per query limit or a coordinator level limit? Maybe, also mention This is coordinator level limit for each pool. Updated comments. http://gerrit.cloudera.org:8080/#/c/21304/17/be/src/service/frontend.cc@103 PS17, Line 103: "to all DBCP connection pools created on the coordinator."); > If -1 is indefinite, 0 means immediately returns error? Yes. Updated the comments. http://gerrit.cloudera.org:8080/#/c/21304/17/common/thrift/ExternalDataSource.thrift File common/thrift/ExternalDataSource.thrift: http://gerrit.cloudera.org:8080/#/c/21304/17/common/thrift/ExternalDataSource.thrift@144 PS17, Line 144: reference count equals 0 > reference count is tracked across all queries for a given data source in a Yes, updated comments. http://gerrit.cloudera.org:8080/#/c/21304/17/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java File fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java: http://gerrit.cloudera.org:8080/#/c/21304/17/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@91 PS17, Line 91: > what's a typical cache key here? Adding comments would be useful. Added comments. http://gerrit.cloudera.org:8080/#/c/21304/17/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@144 PS17, Line 144: } > Some comments explaining when cleanup() is called would be good. Is it at s This function is running on a working thread in the coordinator daemon, clean up idle DataSource in every 10 seconds. Added comments for this function. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 18 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 30 Apr 2024 21:04:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#18). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Note that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes. Flag variable
'dbcp_data_source_idle_timeout_s' is added to make the duration
configurable.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Note that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TPCDS tests for JDBC tables run only for release/exhaustive builds.
TPCH tests for JDBC tables run for core and exhaustive builds, except
Dockerized builds.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core tests.
- Passed query_test/test_tpcds_queries.py in release/exhaustive build.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,948 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/18
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet:
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#17). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TPCDS tests for JDBC tables run only for release/exhaustive builds.
TPCH tests for JDBC tables run for core and exhaustive builds, except
Dockerized builds.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core tests.
- Passed query_test/test_tpcds_queries.py in release/exhaustive build.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,914 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/17
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 17
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous
[Impala-ASF-CR] IMPALA-12997: Use graceful shutdown for query log tests
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21345 ) Change subject: IMPALA-12997: Use graceful shutdown for query log tests .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21345/4/tests/common/custom_cluster_test_suite.py File tests/common/custom_cluster_test_suite.py: http://gerrit.cloudera.org:8080/#/c/21345/4/tests/common/custom_cluster_test_suite.py@222 PS4, Line 222: method > Later lines expect method to not be None, so I suspect this was overly defe Ack -- To view, visit http://gerrit.cloudera.org:8080/21345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc Gerrit-Change-Number: 21345 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2024 17:42:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12997: Use graceful shutdown for query log tests
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21345 ) Change subject: IMPALA-12997: Use graceful shutdown for query log tests .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21345/4/tests/common/custom_cluster_test_suite.py File tests/common/custom_cluster_test_suite.py: http://gerrit.cloudera.org:8080/#/c/21345/4/tests/common/custom_cluster_test_suite.py@222 PS4, Line 222: method method is used before checking if it's not None in next line -- To view, visit http://gerrit.cloudera.org:8080/21345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc Gerrit-Change-Number: 21345 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2024 01:12:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#16). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TPCDS tests for JDBC tables run only for release/exhaustive builds.
TPCH tests for JDBC tables run for core and exhaustive builds, except
Dockerized builds.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core tests.
- Passed query_test/test_tpcds_queries.py in release/exhaustive build.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,914 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/16
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 16
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- TPCDS tests run only for release/exhaustive builds.
TPCH tests run for core builds.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,914 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/15
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 03:10:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 02:08:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 00:14:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Apr 2024 23:37:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21304/6/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/21304/6/be/src/service/frontend.cc@99
PS6, Line 99: DEFINE_int32(dbcp_max_conn_pool_size, 8,
> Ah, that makes it trickier. I assumed the point of this setting is to limit
This parameter limit each connection pool size, e.g. number of concurrent TCP
connections in a pool. We create DBCP connection pool for each unique
combination of database and username, which is implemented by setting cache key
as dataSourceCacheKey = String.format("%s.%s", jdbcUrl, username) in
GenericJdbcDatabaseAccessor.initializeDatabaseSource(). Each DataSource object
corresponds to one connection pool.
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
File
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java:
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@93
PS12, Line 93: synchronized (this) {
> Ah, synchronized methods would have only prevented other calls to the same
'get', 'remove' and 'cleanup' use same synchronized object 'this'. Multiple
threads will not able to run multiple methods on the same synchronized object
simultaneously.
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@148
PS12, Line 148: } catch (InterruptedException e) {
> I think the only exception we'd see here is InterruptedException, which doe
Lower the log level to info
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Mon, 22 Apr 2024 23:26:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#14). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/14
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#13). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/13
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 13
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a > It's complaining that the 'and' is at the end of this line and not the star Will move 'and' to the start of next line. Thanks -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 21:08:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a > flake8: W504 line break after binary operator If adding '\' after 'and', it complains 'the backslash is redundant between brackets'. Ignore this warning. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 19:50:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#12). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/12
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py@791 PS10, Line 791: a > flake8: W504 line break after binary operator done -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 18:57:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#11). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/11
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#10). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/10
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#9). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,890 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/9
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Also fixes a small bug for database type since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,873 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/8
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation.
It fixes the race condition for the caching of SQL DataSource objects by
using new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
java.sql.Connection.close() is not effectively to remove a closed
connection from connection pool, which causes JDBC handler threads to
wait available connections from connection pool for long time. The work
around is to call BasicDataSource.invalidateConnection() API to close a
connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
16 files changed, 1,791 insertions(+), 84 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/6
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#5). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation.
It fixes the race condition for the caching of SQL DataSource objects by
using new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
java.sql.Connection.close() is not effectively to remove a closed
connection from connection pool, which causes JDBC handler threads to
wait available connections from connection pool for long time. The work
around is to call BasicDataSource.invalidateConnection() API to close a
connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
16 files changed, 1,788 insertions(+), 84 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/5
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-13016: Fix ambiguous row regex that check for no-existence
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21333 ) Change subject: IMPALA-13016: Fix ambiguous row_regex that check for no-existence .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21333 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic81de34bf997dfaf1c199b1fe1b05346b55ff4da Gerrit-Change-Number: 21333 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Apr 2024 00:23:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation.
It fixes the race condition for the caching of SQL DataSource objects by
using new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
java.sql.Connection.close() is not effectively to remove a closed
connection from connection pool, which causes JDBC handler threads to
wait available connections from connection pool for long time. The work
around is to call BasicDataSource.invalidateConnection() API to close a
connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in a following
patch.
Testing:
- Passed core-test.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
16 files changed, 1,788 insertions(+), 84 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/4
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore in the web debug endpoint
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21294 ) Change subject: IMPALA-12874: Identify active and standby catalog and statestore in the web debug endpoint .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2 Gerrit-Change-Number: 21294 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 17 Apr 2024 20:21:43 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC
tables
..
WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset
of TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries
for external JDBC tables with Impala-Impala federation.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TODO
- run TPCDS queries in exhaustive mode.
- set proper default values for maxTotal and maxWaitMillis of DBCP
configuration parameters.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching
expected results for some decimal values.
- Coordinator open multiple JDBC connections in parallel for some
complex TPCDS quries with multiple DataSource scan nodes in
query plan, which cause connection not available from connection
pool if maxTotal is less than total number of scan nodes.
- maxWaitMillis is not working.
Testing:
- TODO Pass all TPCH/TCPDS queries.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
10 files changed, 1,532 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/2
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21304
Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC
tables
..
WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset
of TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries
for external JDBC tables with Impala-Impala federation.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TODO
- run TPCDS queries in exhaustive mode.
- set proper default values for maxTotal and maxWaitMillis of DBCP
configuration parameters.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching
expected results for some decimal values.
- Coordinator open multiple JDBC connections in parallel for some
complex TPCDS quries with multiple DataSource scan nodes in
query plan, which cause connection not available from connection
pool if maxTotal is less than total number of scan nodes.
- maxWaitMillis is not working.
Testing:
- TODO Pass all TPCH/TCPDS queries.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
10 files changed, 1,527 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/1
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21294 )
Change subject: IMPALA-12874: Identify active and standby catalog and
statestore the web debug endpoint
..
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc@283
PS1, Line 283: if (metric->GetValue()) {
: document->AddMember(
: "catalogd_active_status", "Active",
document->GetAllocator());
: } else {
: document->AddMember(
: "catalogd_active_status", "Standby",
document->GetAllocator());
: }
> Tried, but the initializer of rapidjson::GenericValue seems not working thi
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2
Gerrit-Change-Number: 21294
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 15 Apr 2024 17:22:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21294 )
Change subject: IMPALA-12874: Identify active and standby catalog and
statestore the web debug endpoint
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21294/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21294/1//COMMIT_MSG@15
PS1, Line 15: active as
: the status
nit: status as active
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc@283
PS1, Line 283: if (metric->GetValue()) {
: document->AddMember(
: "catalogd_active_status", "Active",
document->GetAllocator());
: } else {
: document->AddMember(
: "catalogd_active_status", "Standby",
document->GetAllocator());
: }
nit: this can be simplified as
document->AddMember("catalogd_active_status", (metric->GetValue() ? "Active" :
"Standby"), document->GetAllocator());
--
To view, visit http://gerrit.cloudera.org:8080/21294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2
Gerrit-Change-Number: 21294
Gerrit-PatchSet: 1
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 12 Apr 2024 16:36:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 9: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 10 Apr 2024 22:29:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 17 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 10 Apr 2024 20:14:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21168 )
Change subject: IMPALA-12920: Support ai_generate_text built-in function for
OpenAI's chat completion API
..
Patch Set 16: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21168/16/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:
http://gerrit.cloudera.org:8080/#/c/21168/16/be/src/exprs/expr-test.cc@11341
PS16, Line 11341: string("A null-terminated string is a character string in a
programming "
: "language like C and C++ that ends with a null
character (\'\\0\') . This "
: "character represents the end of the string and is
used to determine the "
: "conclusion of the text. Essentially, it is a
sequence of characters "
: "followed by a null byte.")
nit: use content instead
--
To view, visit http://gerrit.cloudera.org:8080/21168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
Gerrit-Change-Number: 21168
Gerrit-PatchSet: 16
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Wed, 10 Apr 2024 16:57:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12983 (Part 1): Push down simple limit to remote database
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21281 Change subject: WIP IMPALA-12983 (Part 1): Push down simple limit to remote database .. WIP IMPALA-12983 (Part 1): Push down simple limit to remote database This patch pushes down the simple limit on DataSourceScanNode to remote database. Simple limit means the scan node does not have scan conjunct. Testing: - TODO: Add unit-test to verify RowsRead of DataSourceNode in query profiles, which are the rows read from JDBC driver. - Passed end-to-end unit tests for external JDBC tables Change-Id: I139d34cc3b70db14d5f45266d2cbd586a53c8382 --- M be/src/exec/data-source-scan-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java 3 files changed, 9 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/21281/1 -- To view, visit http://gerrit.cloudera.org:8080/21281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I139d34cc3b70db14d5f45266d2cbd586a53c8382 Gerrit-Change-Number: 21281 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 14: (2 comments) http://gerrit.cloudera.org:8080/#/c/21168/14/be/src/exprs/ai-functions.inline.h File be/src/exprs/ai-functions.inline.h: http://gerrit.cloudera.org:8080/#/c/21168/14/be/src/exprs/ai-functions.inline.h@43 PS14, Line 43: const string AiFunctions::AI_GENERATE_TXT_JSON_PARSE_ERROR = "Invalid Json"; : const string AiFunctions::AI_GENERATE_TXT_INVALID_PROTOCOL_ERROR = : "Invalid Protocol, use https"; : const string AiFunctions::AI_GENERATE_TXT_UNSUPPORTED_ENDPOINT_ERROR = : "Unsupported Endpoint"; : const string AiFunctions::AI_GENERATE_TXT_INVALID_PROMPT_ERROR = : "Invalid Prompt, cannot be null or empty"; : const string AiFunctions::AI_GENERATE_TXT_MSG_OVERRIDE_FORBIDDEN_ERROR = : "Invalid override, 'messages' cannot be overriden"; : const string AiFunctions::AI_GENERATE_TXT_N_OVERRIDE_FORBIDDEN_ERROR = : "Invalid override, 'n' must be of integer type and have value 1"; missed static http://gerrit.cloudera.org:8080/#/c/21168/14/be/src/exprs/ai-functions.inline.h@61 PS14, Line 61: string AiFunctions::ai_api_key_; missed static -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 14 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 09 Apr 2024 20:07:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/21168/9/bin/load-data.py File bin/load-data.py: http://gerrit.cloudera.org:8080/#/c/21168/9/bin/load-data.py@320 PS9, Line 320: TESTDATA_JCEKS_DIR > Created the directory under testdata. Need to add the file in bin/rat_exclude_files.txt, or create the directory in runtime. -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 09 Apr 2024 01:14:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12965: Add debug query option RUNTIME FILTER IDS TO SKIP
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21230 ) Change subject: IMPALA-12965: Add debug query option RUNTIME_FILTER_IDS_TO_SKIP .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I897e37685dd1ec279989b55560ec7616a00d2280 Gerrit-Change-Number: 21230 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 09 Apr 2024 00:07:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/21168/9/bin/load-data.py File bin/load-data.py: http://gerrit.cloudera.org:8080/#/c/21168/9/bin/load-data.py@320 PS9, Line 320: TESTDATA_JCEKS_DIR make sure the directory is created -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 08 Apr 2024 23:53:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 08 Apr 2024 20:13:08 + Gerrit-HasComments: No
