[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Reviewed-on: http://gerrit.cloudera.org:8080/17104 Reviewed-by: Thomas Tauber-Marshall Tested-by: Thomas Tauber-Marshall --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 959 insertions(+), 760 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 14 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 12 Mar 2021 17:49:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has removed a vote on this change. Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: verify job was manually aborted so that we can just run it on the tip of the commit chain: https://gerrit.cloudera.org/#/c/17122/ -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 09 Mar 2021 19:03:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6945/ -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 09 Mar 2021 19:02:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6945/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 09 Mar 2021 18:58:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: verify failed due to IMPALA-10376 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 09 Mar 2021 18:57:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6940/ -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 08 Mar 2021 23:30:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6940/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 08 Mar 2021 17:47:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 08 Mar 2021 17:45:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8298/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 04 Mar 2021 13:23:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h@679 PS10, Line 679: external_exec_request is a statement that was prepared by an external frontend using : /// Impala PlanNodes. > or null if the external frontend isn't being used Done http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc@1207 PS10, Line 1207: (*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address()); > I think you misunderstood my previous comment here, sorry if I was unclear. Done http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1080 PS10, Line 1080: false > could you add a "/* must_exist */" before this, for readability, ie: Done -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 04 Mar 2021 13:12:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#13). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 959 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/13 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8291/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 11 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:02:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#11). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 989 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/11 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 11 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h@679 PS10, Line 679: external_exec_request is a statement that was prepared by an external frontend using : /// Impala PlanNodes. or null if the external frontend isn't being used http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc@1207 PS10, Line 1207: (*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address()); I think you misunderstood my previous comment here, sorry if I was unclear. I was suggesting moving all of these calls to set other values on the TExecRequest into SetExternalPlan() (not just the call to set local_time, which you did move). If you do that, then I think you can eliminate most of the extra ClientRequestState functions that you added - SetExecRequestQueryId/SetExecRequestHostname/etc. http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1080 PS10, Line 1080: false could you add a "/* must_exist */" before this, for readability, ie: tbl = getTable(tblName.getDb(), tblName.getTbl(), /* must_exist */ false); here and elsewhere in this patch -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 18:55:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8285/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:54:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8282/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:49:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 7: (13 comments) http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc@67 PS7, Line 67: exec_request > Nit: I think it would be clearer to call this 'external_exec_request' Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc@458 PS7, Line 458: execRequest > nit: exec_request (here and several other places in this patch) Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h@678 PS7, Line 678: /// have been checked out. > comment might benefit from a brief mention of what 'exec_request' is Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@319 PS7, Line 319: "TExecRequest-{internal|external}.{query_id.hi}-{query_id.lo}"); > might be worth explicitly saying this is for debugging Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1114 PS7, Line 1114: if (FLAGS_dump_exec_request_path.empty()) : return; > nit: this can go on one line (and if it couldn't, Impala always uses braces Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1118 PS7, Line 1118: FLAGS_dump_exec_request_path + "/TExecRequest-" + dumpType + "." + : std::to_string(queryID.hi) + "-" + std::to_string(queryID.lo) > For constructing the filename, I think it would be cleaner to use the Subst Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1140 PS7, Line 1140: exec_request > It might be clearer for this to be 'external_exec_request' to emphasize tha Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1191 PS7, Line 1191: exec_request > Nit: From a style point, we prefer explicit checks against nullptr ("exec_r Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1198 PS7, Line 1198: exec_request > Nit: Same here (use explicit nullptr checks) Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1202 PS7, Line 1202: // Update the externally provided exec_request with Impala's query_id() : (*query_handle)->SetExecRequestQueryId(query_id); : // Update coordinator related internal addresses : (*query_handle)->SetExecRequestHostname( : ExecEnv::GetInstance()->configured_backend_address().hostname); : (*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address()); : // Update the field of 'local_time_zone' of 'query_ctx'. : (*query_handle)->SetExecRequestLocalTimeZone(query_ctx.local_time_zone); > might be nice to move this into QueryDriver::SetExternalPlan() Done http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1213 PS7, Line 1213: RETURN_IF_ERROR((*query_handle)->UpdateQueryStatus(exec_status)); > What's the point of this call? I think it will always be called with OK and Removed http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt File common/thrift/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt@70 PS7, Line 70: # Also do not generate ImpalaService.thrift because the generated code doesn't : # compile with hive if the thrift version in hive is 0.9.0 > Nit: We can remove this part of the comment Done http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml@434 PS7, Line 434: org.apache.hive : hive-classification : ${hive.version} > Does this bring in any dependencies that we need to exclude? I checked both and only hive-classification.jar was added -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John She
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#10). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 989 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/10 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/17104/9/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17104/9/be/src/runtime/query-driver.cc@76 PS9, Line 76: exec_request_->query_exec_request.query_ctx.__set_local_time_zone(query_ctx.local_time_zone); line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:29:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#9). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 988 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/9 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8277/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 01:54:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#8). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 986 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/8 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 7: (6 comments) http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc@458 PS7, Line 458: execRequest nit: exec_request (here and several other places in this patch) http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h@678 PS7, Line 678: /// have been checked out. comment might benefit from a brief mention of what 'exec_request' is http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@319 PS7, Line 319: "TExecRequest-{internal|external}.{query_id.hi}-{query_id.lo}"); might be worth explicitly saying this is for debugging http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1114 PS7, Line 1114: if (FLAGS_dump_exec_request_path.empty()) : return; nit: this can go on one line (and if it couldn't, Impala always uses braces around blocks) Some of these formatting issues can be avoided if you check out clang-format-diff, see: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868536 http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1202 PS7, Line 1202: // Update the externally provided exec_request with Impala's query_id() : (*query_handle)->SetExecRequestQueryId(query_id); : // Update coordinator related internal addresses : (*query_handle)->SetExecRequestHostname( : ExecEnv::GetInstance()->configured_backend_address().hostname); : (*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address()); : // Update the field of 'local_time_zone' of 'query_ctx'. : (*query_handle)->SetExecRequestLocalTimeZone(query_ctx.local_time_zone); might be nice to move this into QueryDriver::SetExternalPlan() http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1213 PS7, Line 1213: RETURN_IF_ERROR((*query_handle)->UpdateQueryStatus(exec_status)); What's the point of this call? I think it will always be called with OK and not do anything -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 23:56:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 7: (7 comments) Posting comments that I have so far http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc@67 PS7, Line 67: exec_request Nit: I think it would be clearer to call this 'external_exec_request' http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1118 PS7, Line 1118: FLAGS_dump_exec_request_path + "/TExecRequest-" + dumpType + "." + : std::to_string(queryID.hi) + "-" + std::to_string(queryID.lo) For constructing the filename, I think it would be cleaner to use the Substitute() function that takes a template string and args: https://github.com/apache/impala/blob/master/be/src/gutil/strings/substitute.h#L172 There are several uses in this file to consult. http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1140 PS7, Line 1140: exec_request It might be clearer for this to be 'external_exec_request' to emphasize that this is ordinarily null. http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1191 PS7, Line 1191: exec_request Nit: From a style point, we prefer explicit checks against nullptr ("exec_request != nullptr") rather than implicit non-zero checks of pointer values. http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1198 PS7, Line 1198: exec_request Nit: Same here (use explicit nullptr checks) http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt File common/thrift/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt@70 PS7, Line 70: # Also do not generate ImpalaService.thrift because the generated code doesn't : # compile with hive if the thrift version in hive is 0.9.0 Nit: We can remove this part of the comment http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml@434 PS7, Line 434: org.apache.hive : hive-classification : ${hive.version} Does this bring in any dependencies that we need to exclude? If you haven't already, it's worth checking mvn dependency:tree or diffing the fe/target/build-classpath.txt to see if this is adding anything. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 20:59:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8252/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 15:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#7). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 986 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/7 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8250/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 14:54:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/17104/6/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/17104/6/be/src/service/client-request-state.h@245 PS6, Line 245: /// This helper method updates the local time zone of client's TExecRequest. The intended line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 14:35:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#6). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 985 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/6 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8246/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 25 Feb 2021 23:49:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
j...@cloudera.com has uploaded a new patch set (#5) to the change originally created by Kurt Deschler. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 975 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/5 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8202/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 3 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Feb 2021 04:30:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#3). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 943 insertions(+), 755 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/3 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 3 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8186/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 21 Feb 2021 22:06:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello j...@cloudera.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#2). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 20 files changed, 950 insertions(+), 736 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/2 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8185/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 21 Feb 2021 21:49:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 1: (6 comments) http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/runtime/query-driver.cc@66 PS1, Line 66: Status QueryDriver::SetExternalPlan(const TQueryCtx& query_ctx, const TExecRequest& exec_request) { line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/service/impala-server.cc@316 PS1, Line 316: "If set, dump TExecRequest structures to {dump_exec_request_path}/TExecRequest-{internal|external}.{query_id.hi}-{query_id.lo}"); line too long (133 > 90) http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/service/impala-server.cc@1101 PS1, Line 1101: Status status = ExecuteInternal(*query_ctx, execRequest, session_state, ®istered_query, line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/service/impala-server.cc@1189 PS1, Line 1189: // Takes the TQueryCtx and calls into the frontend to initialize the TExecRequest for line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/17104/1/be/src/service/impala-server.cc@1197 PS1, Line 1197: RETURN_IF_ERROR(query_handle->query_driver()->SetExternalPlan(query_ctx, *exec_request)); line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/17104/1/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/17104/1/common/thrift/ImpalaService.thrift@831 PS1, Line 831: TCLIService.TExecuteStatementResp ExecutePlannedStatement(1:TExecutePlannedStatementReq req); line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 21 Feb 2021 21:38:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello j...@cloudera.com, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/17104 to review the following change. Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 20 files changed, 945 insertions(+), 735 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/1 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anonymous Coward