[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 07:53:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6765/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 07:53:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10375: Lock down which filesystems use the file handle cache
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16856 ) Change subject: IMPALA-10375: Lock down which filesystems use the file handle cache .. Patch Set 1: (1 comment) I suggested a refactoring, it would make the patch a bit bigger but might pay off. LMK what you think. http://gerrit.cloudera.org:8080/#/c/16856/1/be/src/runtime/io/scan-range.cc File be/src/runtime/io/scan-range.cc: http://gerrit.cloudera.org:8080/#/c/16856/1/be/src/runtime/io/scan-range.cc@200 PS1, Line 200: if (is_file_handle_caching_enabled() && filesystem_supports_handle_caching(file()) && I was originally thinking about this from the point of view of performance overhead and mostly convinced myself that the extra string parsing wouldn't add much. But it occurs to me that the code would probably be simplified overall if we figured out the filesystem implementation once per scan range, i.e. at the same time as AssignDiskQueue() is called. We could convert it to an enum like FsType in the frontend these disk_id/filename checks in ScanRange would be a bit less ad-hoc. I think that would make handling https://issues.apache.org/jira/browse/HDFS-15289 simpler too once we get to that. -- To view, visit http://gerrit.cloudera.org:8080/16856 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I136c3da9d19590cdbe8623d22480b8dd07192ce3 Gerrit-Change-Number: 16856 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 05:16:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ IMPALA-9930 introduces a new data structure to AdmissionController: 'queue_nodes_'. The intention was that access to 'queue_nodes_' should always be protected by 'queue_nodes_lock_', but this was not always the case. This patch fixes the issue by ensuring that 'queue_nodes_lock_' is held everywhere that 'queue_nodes_' is used. Testing: - Ran TSAN and confirmed there is no longer a data race being reported. Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Reviewed-on: http://gerrit.cloudera.org:8080/16835 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc 1 file changed, 8 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 05:03:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 04:59:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6764/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 04:59:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 5: Hit IMPALA-10066 -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 04:58:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-9865: part 1: basic profile log parser .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10375: Lock down which filesystems use the file handle cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16856 ) Change subject: IMPALA-10375: Lock down which filesystems use the file handle cache .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6762/ -- To view, visit http://gerrit.cloudera.org:8080/16856 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I136c3da9d19590cdbe8623d22480b8dd07192ce3 Gerrit-Change-Number: 16856 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 11 Dec 2020 02:38:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/16855 )
Change subject: IMPALA-10390: impala-profile-tool JSON output
..
Patch Set 3: Code-Review+1
(1 comment)
Thank you, Tim. LGTM!
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc
File be/src/util/impala-profile-tool.cc:
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc@120
PS2, Line 120: if (profile_format == "text") {
> The one line per document is pretty common and is a bit easier to handle wi
Ah, didn't now there is a .jsonl format. I learn new thing.
--
To view, visit http://gerrit.cloudera.org:8080/16855
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67
Gerrit-Change-Number: 16855
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 11 Dec 2020 02:10:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10211 (Part 1): Add support for role-related statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16837 ) Change subject: IMPALA-10211 (Part 1): Add support for role-related statements .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6761/ -- To view, visit http://gerrit.cloudera.org:8080/16837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b204e62a1d8ae1932d955b4efc28be22202860 Gerrit-Change-Number: 16837 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 11 Dec 2020 01:58:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16855 ) Change subject: IMPALA-10390: impala-profile-tool JSON output .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7837/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 01:51:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16855 )
Change subject: IMPALA-10390: impala-profile-tool JSON output
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc
File be/src/util/impala-profile-tool.cc:
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc@44
PS2, Line 44: " json: output as JSON with one profile per line\n"
> missing description for prettyjson option?
Done
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc@120
PS2, Line 120: CHECK_EQ("json", profile_format);
> For json and prettyjson format, what if we return array of json profile?
The one line per document is pretty common and is a bit easier to handle with
command line tools - I discovered there's even a website https://jsonlines.org/
I take your point for the pretty-printed version - it is hard to parse apart
the profiles with them split across lines, so I did convert that into an array
representation.
--
To view, visit http://gerrit.cloudera.org:8080/16855
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67
Gerrit-Change-Number: 16855
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 11 Dec 2020 01:29:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Hello Riza Suminto, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16855 to look at the new patch set (#3). Change subject: IMPALA-10390: impala-profile-tool JSON output .. IMPALA-10390: impala-profile-tool JSON output Add --profile_format option that takes options "text", "json" or "prettyjson". "json" and "prettyjson" output the JSON representation of each profile in a dense single-line form and in a human-readable multi-line form respectively. Also implement usage output when --help is passed in. Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 --- M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/util/impala-profile-tool.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h 5 files changed, 99 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/16855/3 -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6760/ -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 11 Dec 2020 01:19:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10375: Lock down which filesystems use the file handle cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16856 ) Change subject: IMPALA-10375: Lock down which filesystems use the file handle cache .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7836/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16856 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I136c3da9d19590cdbe8623d22480b8dd07192ce3 Gerrit-Change-Number: 16856 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 23:51:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6763/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 23:31:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 23:31:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 3: verify failed due to unrelated IMPALA-10066 -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 23:30:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10375: Lock down which filesystems use the file handle cache
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16856 Change subject: IMPALA-10375: Lock down which filesystems use the file handle cache .. IMPALA-10375: Lock down which filesystems use the file handle cache The logic for determining whether to use the file handle cache currently use the file handle cache for anything that is expected to be local. This adds defensive code to limit the file handle cache to filesystems that are known to support it (currently HDFS, S3A, ABFS). This prevents any weird behavior for untested configurations (Alluxio, Ozone) that might be considered local. Testing: - Ran core job Change-Id: I136c3da9d19590cdbe8623d22480b8dd07192ce3 --- M be/src/runtime/io/disk-io-mgr-internal.h M be/src/runtime/io/scan-range.cc 2 files changed, 15 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/16856/1 -- To view, visit http://gerrit.cloudera.org:8080/16856 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I136c3da9d19590cdbe8623d22480b8dd07192ce3 Gerrit-Change-Number: 16856 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10336: Coordinator return incorrect error to client
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16849 ) Change subject: IMPALA-10336: Coordinator return incorrect error to client .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7835/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16849 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75f252e43006c6ff6980800e3254672de396b318 Gerrit-Change-Number: 16849 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 10 Dec 2020 23:22:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6759/ -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 23:16:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10391: Fix LIRS edge case for single unprotected entry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16857 ) Change subject: IMPALA-10391: Fix LIRS edge case for single unprotected entry .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d21b619811a1a7baab1a92790f2ffc03e949131 Gerrit-Change-Number: 16857 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 23:06:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10336: Coordinator return incorrect error to client
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16849 ) Change subject: IMPALA-10336: Coordinator return incorrect error to client .. IMPALA-10336: Coordinator return incorrect error to client Due to race condition, coordinator could set execution status as RPC aborted due to cancellation. This internal error should not be returned to client. Instead of separate function call to get backend status, add more parameters in function BackendState::ApplyExecStatusReport() to return backend status. Testing: - Could not re-produce the issue by running the test case test_scanners.py::TestOrc::test_type_conversions_hive3 in a loop. - Passed exhausive test. Change-Id: I75f252e43006c6ff6980800e3254672de396b318 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc 3 files changed, 23 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16849/2 -- To view, visit http://gerrit.cloudera.org:8080/16849 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I75f252e43006c6ff6980800e3254672de396b318 Gerrit-Change-Number: 16849 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10391: Fix LIRS edge case for single unprotected entry
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16857 Change subject: IMPALA-10391: Fix LIRS edge case for single unprotected entry .. IMPALA-10391: Fix LIRS edge case for single unprotected entry When an unprotected entry is not in the recency list, a lookup will cause it to be moved to be the newest entry in the unprotected list. The fix for IMPALA-10127 introduced a regression when this happens when there is exactly on entry in the unprotected list. The code currently calls RemoveFromUnprotectedList() followed by AddToUnprotectedList(). This now fails because it is doing these operations without manipulating the num_unprotected_ count. RemoveFromUnprotectedList() clears out unprotected_list_front_, because num_unprotected_ is 1. However, AddToUnprotectedList() does not set it back, because it only does that if num_unprotected_ is 0, and the count is not changing. This skips the remove/add in this case if there is exactly one unprotected entry in the list. Testing: - Added a backend test for this specific case and verfied that it failed before the fix and passes now Change-Id: I8d21b619811a1a7baab1a92790f2ffc03e949131 --- M be/src/util/cache/lirs-cache-test.cc M be/src/util/cache/lirs-cache.cc 2 files changed, 43 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/16857/1 -- To view, visit http://gerrit.cloudera.org:8080/16857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8d21b619811a1a7baab1a92790f2ffc03e949131 Gerrit-Change-Number: 16857 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command flakiness
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/16847 ) Change subject: IMPALA-10066: Fix test_cancellation_mid_command flakiness .. Patch Set 2: Code-Review+1 Thanks for the fix! I like the new commit message much better. Carry and concur with +1 from Gabor -- To view, visit http://gerrit.cloudera.org:8080/16847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief384ce59f3ce24f1ab2dfb5fbaf7c9a39b434e0 Gerrit-Change-Number: 16847 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 22:21:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 ) Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6757/ -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 21:57:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/16855 )
Change subject: IMPALA-10390: impala-profile-tool JSON output
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc
File be/src/util/impala-profile-tool.cc:
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc@44
PS2, Line 44: " json: output as JSON with one profile per line\n"
missing description for prettyjson option?
http://gerrit.cloudera.org:8080/#/c/16855/2/be/src/util/impala-profile-tool.cc@120
PS2, Line 120: CHECK_EQ("json", profile_format);
For json and prettyjson format, what if we return array of json profile?
So if this parser return multiple profile, the output is still a valid json and
can be piped to other program that read json like jq.
--
To view, visit http://gerrit.cloudera.org:8080/16855
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67
Gerrit-Change-Number: 16855
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Thu, 10 Dec 2020 21:16:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16855 ) Change subject: IMPALA-10390: impala-profile-tool JSON output .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7832/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 20:34:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16855 ) Change subject: IMPALA-10390: impala-profile-tool JSON output .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7833/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 20:33:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10211 (Part 1): Add support for role-related statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16837 ) Change subject: IMPALA-10211 (Part 1): Add support for role-related statements .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6761/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b204e62a1d8ae1932d955b4efc28be22202860 Gerrit-Change-Number: 16837 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Dec 2020 20:27:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10358: Correct Iceberg type mappings
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16851 ) Change subject: IMPALA-10358: Correct Iceberg type mappings .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16851/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16851/1//COMMIT_MSG@13 PS1, Line 13: * annotates strings with UTF8 in Parquet metadata These changes make sense to me. I did add a TYPE_FIXED_UDA_INTERMEDIATE type internally that has the same semantics as FIXED, i.e. essentially the BINARY variant of CHAR, so that could maybe be something that is extended in future to be a user-visible type. I'm curious how timestamptz works now - do we just treat it as an Impala TIMESTAMP and rely on the Parquet reader to convert when reading? I guess we can't write it if there's a timestamptz column in an Iceberg table? -- To view, visit http://gerrit.cloudera.org:8080/16851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 Gerrit-Change-Number: 16851 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 20:21:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. IMPALA-10343: increase control_service_queue_mem_limit --control_service_queue_mem_limit is set to 1% by default to increase the maximum size of the queue in typical production deployments. E.g. an Impala daemon with a 50GB memory limit will have a limit of 512MB on the control service queue. Add --control_service_queue_mem_limit_floor_bytes so that this does not have the unintended effect of reducing the memory given to the control service queue. I.e. the default behaviour does not change for impala daemons with a daemon mem limit of <= 5000MB, but does increase the control service queue memory limit for impala daemons with mem limits > 5000MB. The default process memory limit in the mocked backend test ExecEnv is changed to be 8GB. Previously it was unlimited, so we couldn't calculate 1% of it. It cannot be unlimited in an actual impalad since IMPALA-5653 was fixed. Testing: This had been previously problematic on a 64 node TPC-DS workload with mt_dop=12 where impalads had ~100GB of memory. Status report RPCs would fail and have to be retried. We tested this new value on the same workload and the retries were avoided. Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Reviewed-on: http://gerrit.cloudera.org:8080/16848 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Tauber-Marshall --- M be/src/runtime/test-env.h M be/src/service/control-service.cc M tests/custom_cluster/test_rpc_timeout.py 3 files changed, 12 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Tauber-Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Hello Riza Suminto, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16855 to look at the new patch set (#2). Change subject: IMPALA-10390: impala-profile-tool JSON output .. IMPALA-10390: impala-profile-tool JSON output Add --profile_format option that takes options "text", "json" or "prettyjson". "json" and "prettyjson" output the JSON representation of each profile in a dense single-line form and in a human-readable multi-line form respectively. Also implement usage output when --help is passed in. Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 --- M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/util/impala-profile-tool.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h 5 files changed, 93 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/16855/2 -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 5: THanks, yeah, there's lots to do. I just added a JSON output here - https://gerrit.cloudera.org/#/c/16855/ and cleaned up some of the --help usage stuff. -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 20:10:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16855 ) Change subject: IMPALA-10390: impala-profile-tool JSON output .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16855/1/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/16855/1/be/src/service/impala-hs2-server.cc@1061 PS1, Line 1061: RuntimeProfile::JsonProfileToStringProfile(*profile.json_output, profile.string_output); line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 20:09:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10390: impala-profile-tool JSON output
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16855 Change subject: IMPALA-10390: impala-profile-tool JSON output .. IMPALA-10390: impala-profile-tool JSON output Add --profile_format option that takes options "text", "json" or "prettyjson". "json" and "prettyjson" output the JSON representation of each profile in a dense single-line form and in a human-readable multi-line form respectively. Also implement usage output when --help is passed in. Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 --- M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/util/impala-profile-tool.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h 5 files changed, 93 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/16855/1 -- To view, visit http://gerrit.cloudera.org:8080/16855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I82ae0fe9379b7e3cbe93166adaa4c37212ea0f67 Gerrit-Change-Number: 16855 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6760/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 19:47:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 19:47:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9865: part 1: basic profile log parser
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16821 ) Change subject: IMPALA-9865: part 1: basic profile log parser .. Patch Set 4: Code-Review+2 This is really cool! I'm already thinking about extensions to the behavior, and I think this will be very useful. Great to get a basic version in first. -- To view, visit http://gerrit.cloudera.org:8080/16821 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6178399ac96e176f7067cc47347e51cda2f3 Gerrit-Change-Number: 16821 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 19:45:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. IMPALA-10361: Use field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. Currently, we use field id to resolve column for Iceberg tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid for Iceberg tables. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Reviewed-on: http://gerrit.cloudera.org:8080/16788 Tested-by: Impala Public Jenkins Reviewed-by: Zoltan Borok-Nagy --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/query-options-test.cc M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/1-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/6-6-305c9b7a-f42d-4245-b806-dfa7a792593f-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/9-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/4-4-7a1a8e89-8aeb-4405-be64-76557432cf21-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-0.parquet A
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 19:00:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10211 (Part 1): Add support for role-related statements
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16837 ) Change subject: IMPALA-10211 (Part 1): Add support for role-related statements .. Patch Set 4: (5 comments) Great work, especially the workarounds for the Ranger bugs! I ran through most of the code, I plan to do another pass soon. http://gerrit.cloudera.org:8080/#/c/16837/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/16837/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@105 PS4, Line 105: issue Is there a Ranger ticket for this? http://gerrit.cloudera.org:8080/#/c/16837/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@133 PS4, Line 133: // actually revoke the role from the group. This should be considered a bug of Is there a Ranger ticket for this? http://gerrit.cloudera.org:8080/#/c/16837/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@143 PS4, Line 143: plugin_.get().revokeRole(request, null); : plugin_.get().grantRole(request, null); Is it possible to run this with more than one threads at the same time? Two parallel grants to the same group could run like this: revoke role // revoking role if the group already had it revoke role // no effect grant role // granting role grant role // revoking role Even if this is possible, I don't think that it is a very serious issue, but it would be good to know whether we have to think about parallelism. http://gerrit.cloudera.org:8080/#/c/16837/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@167 PS4, Line 167: dropping Are we logging "dropping" intentionally? http://gerrit.cloudera.org:8080/#/c/16837/4/testdata/workloads/functional-query/queries/QueryTest/grant_revoke.test File testdata/workloads/functional-query/queries/QueryTest/grant_revoke.test: http://gerrit.cloudera.org:8080/#/c/16837/4/testdata/workloads/functional-query/queries/QueryTest/grant_revoke.test@1277 PS4, Line 1277: # Clean up the granted privileges and test roles. Will it cause problems if we fail do drop these? My understanding is that executing test files stop at the first failed test. -- To view, visit http://gerrit.cloudera.org:8080/16837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2b204e62a1d8ae1932d955b4efc28be22202860 Gerrit-Change-Number: 16837 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Dec 2020 18:43:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/16755 ) Change subject: IMPALA-9856: Enable result spooling by default. .. Patch Set 3: (10 comments) Patch set 3 is a rebase after several commits get in, including IMPALA-10337. I add more descriptive comments on tests where we need to explicitly disable result spooling. We still need to wait for IMPALA-10371 to be resolved before we can enable result spooling by default. http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_admission_controller.py@337 PS1, Line 337: _timeout_ > nit: mention briefly what we are trying to assert, that way we know from th Done http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_admission_controller.py@873 PS1, Line 873: u > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_admission_controller.py@883 PS2, Line 883: : > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_observability.py File tests/custom_cluster/test_observability.py: http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_observability.py@37 PS1, Line 37: ue to unresolve > nit: mention why will this cause a crash Done http://gerrit.cloudera.org:8080/#/c/16755/1/tests/custom_cluster/test_observability.py@38 PS1, Line 38: l > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_observability.py File tests/custom_cluster/test_observability.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_observability.py@40 PS2, Line 40: : > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/1/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16755/1/tests/query_test/test_udfs.py@623 PS1, Line 623: l > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/1/tests/query_test/test_udfs.py@630 PS1, Line 630: > flake8: E501 line too long (98 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py@625 PS2, Line 625: : > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py@632 PS2, Line 632: > flake8: E501 line too long (98 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/16755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e360c1428676d8f3fab5d95efee18aca085eba4 Gerrit-Change-Number: 16755 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 18:38:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16755 ) Change subject: IMPALA-9856: Enable result spooling by default. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7831/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e360c1428676d8f3fab5d95efee18aca085eba4 Gerrit-Change-Number: 16755 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 18:25:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Hello Quanlong Huang, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16755 to look at the new patch set (#3). Change subject: IMPALA-9856: Enable result spooling by default. .. IMPALA-9856: Enable result spooling by default. Result spooling has been relatively stable since it was introduced, and it has several benefits described in IMPALA-8656. This patch enable result spooling (SPOOL_QUERY_RESULTS) query options by default. Furthermore, some tests need to be adjusted to account for result spooling by default. The following are the adjustment categories and list of tests that fall under such category. Change in assertions: PlannerTest#testAcidTableScans PlannerTest#testBloomFilterAssignment PlannerTest#testConstantFolding PlannerTest#testFkPkJoinDetection PlannerTest#testFkPkJoinDetectionWithHDFSNumRowsEstDisabled PlannerTest#testKuduSelectivity PlannerTest#testMaxRowSize PlannerTest#testMinMaxRuntimeFilters PlannerTest#testMinMaxRuntimeFiltersWithHDFSNumRowsEstDisabled PlannerTest#testMtDopValidation PlannerTest#testParquetFiltering PlannerTest#testParquetFilteringDisabled PlannerTest#testPartitionPruning PlannerTest#testPreaggBytesLimit PlannerTest#testResourceRequirements PlannerTest#testRuntimeFilterQueryOptions PlannerTest#testSortExprMaterialization PlannerTest#testSpillableBufferSizing PlannerTest#testTableSample PlannerTest#testTpch PlannerTest#testKuduTpch PlannerTest#testTpchNested PlannerTest#testUnion TpcdsPlannerTest custom_cluster/test_admission_controller.py::TestAdmissionController::test_dedicated_coordinator_planner_estimates custom_cluster/test_admission_controller.py::TestAdmissionController::test_memory_rejection custom_cluster/test_admission_controller.py::TestAdmissionController::test_pool_mem_limit_configs metadata/test_explain.py::TestExplain::test_explain_level2 metadata/test_explain.py::TestExplain::test_explain_level3 metadata/test_stats_extrapolation.py::TestStatsExtrapolation::test_stats_extrapolation Increase BUFFER_POOL_LIMIT: query_test/test_queries.py::TestQueries::test_analytic_fns query_test/test_runtime_filters.py::TestRuntimeRowFilters::test_row_filter_reservation query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits_full_output query_test/test_spilling.py::TestSpillingBroadcastJoins::test_spilling_broadcast_joins query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive query_test/test_udfs.py::TestUdfExecution::test_mem_limits Increase MEM_LIMIT: query_test/test_mem_usage_scaling.py::TestExchangeMemUsage::test_exchange_mem_usage_scaling query_test/test_mem_usage_scaling.py::TestScanMemLimit::test_hdfs_scanner_thread_mem_scaling Increase MAX_ROW_SIZE: custom_cluster/test_parquet_max_page_header.py::TestParquetMaxPageHeader::test_large_page_header_config query_test/test_insert.py::TestInsertQueries::test_insert_large_string query_test/test_query_mem_limit.py::TestQueryMemLimit::test_mem_limit query_test/test_scanners.py::TestTextSplitDelimiters::test_text_split_across_buffers_delimiter query_test/test_scanners.py::TestWideRow::test_wide_row Disable result spooling to maintain assertion: custom_cluster/test_admission_controller.py::TestAdmissionController::test_set_request_pool custom_cluster/test_admission_controller.py::TestAdmissionController::test_timeout_reason_host_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_timeout_reason_pool_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_queue_reasons_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_pool_config_change_while_queued custom_cluster/test_query_retries.py::TestQueryRetries::test_retry_fetched_rows custom_cluster/test_query_retries.py::TestQueryRetries::test_retry_finished_query custom_cluster/test_scratch_disk.py::TestScratchDir::test_no_dirs custom_cluster/test_scratch_disk.py::TestScratchDir::test_non_existing_dirs custom_cluster/test_scratch_disk.py::TestScratchDir::test_non_writable_dirs query_test/test_insert.py::TestInsertQueries::test_insert_large_string (the last query only) query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan query_test/test_mem_usage_scaling.py::TestScanMemLimit::test_kudu_scan_mem_usage query_test/test_queries.py::TestQueriesParquetTables::test_very_large_strings query_test/test_query_mem_limit.py::TestCodegenMemLimit::test_codegen_mem_limit shell/test_shell_client.py::TestShellClient::test_fetch_size Disable result spooling to avoid crash (IMPALA-10371): custom_cluster/test_observability.py::TestObservability::test_host_profile_jvm_gc_metrics query_test/test_udfs.py::TestUdfExecution::test_java_udfs query_test/test_udfs.py::TestUdfTargeted::test_udf_profile
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 18:01:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 17:56:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6759/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 17:47:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 17:47:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10383: Fix data race on AdmissionController::queue nodes
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16835 ) Change subject: IMPALA-10383: Fix data race on AdmissionController::queue_nodes_ .. Patch Set 2: verify appears to have hit IMPALA-9049, running again -- To view, visit http://gerrit.cloudera.org:8080/16835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05b70112300d3a886bdd5f02fda0bd7ca98df6cf Gerrit-Change-Number: 16835 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 17:46:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 3: Code-Review+2 Thanks for taking care of this! -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 17:43:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16755 ) Change subject: IMPALA-9856: Enable result spooling by default. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7830/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e360c1428676d8f3fab5d95efee18aca085eba4 Gerrit-Change-Number: 16755 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 17:41:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16755 ) Change subject: IMPALA-9856: Enable result spooling by default. .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_admission_controller.py@883 PS2, Line 883: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_observability.py File tests/custom_cluster/test_observability.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/custom_cluster/test_observability.py@40 PS2, Line 40: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py@625 PS2, Line 625: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/16755/2/tests/query_test/test_udfs.py@632 PS2, Line 632: _ flake8: E501 line too long (98 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e360c1428676d8f3fab5d95efee18aca085eba4 Gerrit-Change-Number: 16755 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 10 Dec 2020 17:20:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9856: Enable result spooling by default.
Hello Quanlong Huang, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16755 to look at the new patch set (#2). Change subject: IMPALA-9856: Enable result spooling by default. .. IMPALA-9856: Enable result spooling by default. Result spooling has been relatively stable since it was introduced, and it has several benefits described in IMPALA-8656. This patch enable result spooling (SPOOL_QUERY_RESULTS) query options by default. Furthermore, some tests need to be adjusted to account for result spooling by default. The following are the adjustment categories and list of tests that fall under such category. Change in assertions: PlannerTest#testAcidTableScans PlannerTest#testBloomFilterAssignment PlannerTest#testConstantFolding PlannerTest#testFkPkJoinDetection PlannerTest#testFkPkJoinDetectionWithHDFSNumRowsEstDisabled PlannerTest#testKuduSelectivity PlannerTest#testMaxRowSize PlannerTest#testMinMaxRuntimeFilters PlannerTest#testMinMaxRuntimeFiltersWithHDFSNumRowsEstDisabled PlannerTest#testMtDopValidation PlannerTest#testParquetFiltering PlannerTest#testParquetFilteringDisabled PlannerTest#testPartitionPruning PlannerTest#testPreaggBytesLimit PlannerTest#testResourceRequirements PlannerTest#testRuntimeFilterQueryOptions PlannerTest#testSortExprMaterialization PlannerTest#testSpillableBufferSizing PlannerTest#testTableSample PlannerTest#testTpch PlannerTest#testKuduTpch PlannerTest#testTpchNested PlannerTest#testUnion TpcdsPlannerTest custom_cluster/test_admission_controller.py::TestAdmissionController::test_dedicated_coordinator_planner_estimates custom_cluster/test_admission_controller.py::TestAdmissionController::test_memory_rejection custom_cluster/test_admission_controller.py::TestAdmissionController::test_pool_mem_limit_configs metadata/test_explain.py::TestExplain::test_explain_level2 metadata/test_explain.py::TestExplain::test_explain_level3 metadata/test_stats_extrapolation.py::TestStatsExtrapolation::test_stats_extrapolation Increase BUFFER_POOL_LIMIT: query_test/test_queries.py::TestQueries::test_analytic_fns query_test/test_runtime_filters.py::TestRuntimeRowFilters::test_row_filter_reservation query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits_full_output query_test/test_spilling.py::TestSpillingBroadcastJoins::test_spilling_broadcast_joins query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive query_test/test_udfs.py::TestUdfExecution::test_mem_limits Increase MEM_LIMIT: query_test/test_mem_usage_scaling.py::TestExchangeMemUsage::test_exchange_mem_usage_scaling query_test/test_mem_usage_scaling.py::TestScanMemLimit::test_hdfs_scanner_thread_mem_scaling Increase MAX_ROW_SIZE: custom_cluster/test_parquet_max_page_header.py::TestParquetMaxPageHeader::test_large_page_header_config query_test/test_insert.py::TestInsertQueries::test_insert_large_string query_test/test_query_mem_limit.py::TestQueryMemLimit::test_mem_limit query_test/test_scanners.py::TestTextSplitDelimiters::test_text_split_across_buffers_delimiter query_test/test_scanners.py::TestWideRow::test_wide_row Disable result spooling to maintain assertion: custom_cluster/test_admission_controller.py::TestAdmissionController::test_set_request_pool custom_cluster/test_admission_controller.py::TestAdmissionController::test_timeout_reason_host_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_timeout_reason_pool_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_queue_reasons_memory custom_cluster/test_admission_controller.py::TestAdmissionController::test_pool_config_change_while_queued custom_cluster/test_query_retries.py::TestQueryRetries::test_retry_fetched_rows custom_cluster/test_query_retries.py::TestQueryRetries::test_retry_finished_query custom_cluster/test_scratch_disk.py::TestScratchDir::test_no_dirs custom_cluster/test_scratch_disk.py::TestScratchDir::test_non_existing_dirs custom_cluster/test_scratch_disk.py::TestScratchDir::test_non_writable_dirs query_test/test_insert.py::TestInsertQueries::test_insert_large_string (the last query only) query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan query_test/test_mem_usage_scaling.py::TestScanMemLimit::test_kudu_scan_mem_usage query_test/test_queries.py::TestQueriesParquetTables::test_very_large_strings query_test/test_query_mem_limit.py::TestCodegenMemLimit::test_codegen_mem_limit shell/test_shell_client.py::TestShellClient::test_fetch_size Disable result spooling to avoid crash (IMPALA-10371): custom_cluster/test_observability.py::TestObservability::test_host_profile_jvm_gc_metrics query_test/test_udfs.py::TestUdfExecution::test_java_udfs query_test/test_udfs.py::TestUdfTargeted::test_udf_profile
[Impala-ASF-CR] IMPALA-10358: Correct Iceberg type mappings
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16851 ) Change subject: IMPALA-10358: Correct Iceberg type mappings .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7829/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 Gerrit-Change-Number: 16851 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 17:12:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10358: Correct Iceberg type mappings
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16851 ) Change subject: IMPALA-10358: Correct Iceberg type mappings .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/16851/1/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/16851/1/tests/query_test/test_iceberg.py@20 PS1, Line 20: import tempfile flake8: F401 'tempfile' imported but unused http://gerrit.cloudera.org:8080/#/c/16851/1/tests/query_test/test_iceberg.py@119 PS1, Line 119: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/16851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 Gerrit-Change-Number: 16851 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 16:51:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10358: Correct Iceberg type mappings
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16851 Change subject: IMPALA-10358: Correct Iceberg type mappings .. IMPALA-10358: Correct Iceberg type mappings The Iceberg format spec defines what types to use for different file formats, e.g.: https://iceberg.apache.org/spec/#parquet Impala should follow the specification, so this patch * annotates strings with UTF8 in Parquet metadata * removes fixed(L) <-> CHAR(L) mapping This patch also refactors the type/schema conversions as Impala => Iceberg conversions were duplicated in IcebergCatalogOpExecutor and IcebergUtil. I introduced the class 'IcebergSchemaConverter' to contain the code for conversions. Testing: * added test to check CHAR and VARCHAR types are not allowed * added test to check that strings are annotated with UTF8 Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M tests/query_test/test_iceberg.py 11 files changed, 257 insertions(+), 271 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/16851/1 -- To view, visit http://gerrit.cloudera.org:8080/16851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 Gerrit-Change-Number: 16851 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 3 Gerrit-Owner: liuyao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 16:32:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6754/ -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 16:05:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 13:27:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16850 to look at the new patch set (#3). Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. IMPALA-10384: Make partition names consistent between BE and FE In the BE we build partition names with the trailing char '/'. In the FE we build partition names without a trailing char. We should make this consistent because this causes some annoying string adjustments in the FE and can cause hidden bugs. This patch creates partition names without the trailing '/' both in the BE and the FE. This follows Hive's behavior that also prints partition names without the trailing '/'. Testing: * Ran exhaustive tests Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 --- M be/src/exec/hdfs-table-sink.cc M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionSdCompareTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M testdata/workloads/functional-query/queries/QueryTest/alter-table.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test M testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test M testdata/workloads/functional-query/queries/QueryTest/create-table.test M testdata/workloads/functional-query/queries/QueryTest/date-fileformat-support.test M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test M testdata/workloads/functional-query/queries/QueryTest/hdfs-caching.test M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/insert_null.test M testdata/workloads/functional-query/queries/QueryTest/insert_overwrite.test M testdata/workloads/functional-query/queries/QueryTest/insert_part_key.test M testdata/workloads/functional-query/queries/QueryTest/insert_permutation.test M testdata/workloads/functional-query/queries/QueryTest/load.test M testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test M testdata/workloads/tpcds-insert/queries/partitioned-insert.test 22 files changed, 740 insertions(+), 747 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16850/3 -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7827/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 12:51:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command flakiness
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/16847 ) Change subject: IMPALA-10066: Fix test_cancellation_mid_command flakiness .. Patch Set 2: Thank you for the review Gabor, I ended up rewriting the commit message, it looks better now. -- To view, visit http://gerrit.cloudera.org:8080/16847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief384ce59f3ce24f1ab2dfb5fbaf7c9a39b434e0 Gerrit-Change-Number: 16847 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 12:43:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command flakiness
Hello Gabor Kaszab, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16847 to look at the new patch set (#2). Change subject: IMPALA-10066: Fix test_cancellation_mid_command flakiness .. IMPALA-10066: Fix test_cancellation_mid_command flakiness This change adds additional synchronisation to fix the flaky test. The test failures were happening because the test did not wait for the output of the SIGINT (^C) to arrive. When this was delayed it cluttered the impala-shell output and other expect calls could fail. Testing: - executed the test locally 250 times without failures, without this fix there were about 3 failures in a 100 execution Change-Id: Ief384ce59f3ce24f1ab2dfb5fbaf7c9a39b434e0 --- M tests/shell/test_shell_interactive.py 1 file changed, 15 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/16847/2 -- To view, visit http://gerrit.cloudera.org:8080/16847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ief384ce59f3ce24f1ab2dfb5fbaf7c9a39b434e0 Gerrit-Change-Number: 16847 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
..
Patch Set 7:
(3 comments)
Thanks for new feature, Zoltan. Just some nits.
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG@7
PS4, Line 7: identity-partitioned Iceberg tables
Shall we use "Iceberg tables with only 'IDENTITY' partition" here?
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift@194
PS4, Line 194: tional string tab
Maybe 'TIcebergOperationParam' here? These variable is not operation itself,
just related to operation.
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4655
PS4, Line 4655: ToCreate.isEmpty()) {
It there any difference between 'partName' and partName.substring(0,
partName.length())?
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Thu, 10 Dec 2020 12:41:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command flakiness
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16847 ) Change subject: IMPALA-10066: Fix test_cancellation_mid_command flakiness .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief384ce59f3ce24f1ab2dfb5fbaf7c9a39b434e0 Gerrit-Change-Number: 16847 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 12:30:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6758/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 12:28:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
wangsheng has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. IMPALA-10361: Use field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. Currently, we use field id to resolve column for Iceberg tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid for Iceberg tables. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/query-options-test.cc M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/1-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/6-6-305c9b7a-f42d-4245-b806-dfa7a792593f-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/9-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/4-4-7a1a8e89-8aeb-4405-be64-76557432cf21-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-0.parquet A
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 ) Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7826/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 12:16:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 ) Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 12:08:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 ) Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. Patch Set 7: PS7 is a rebase that resolves the merge conflict. -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 11:57:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 ) Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6757/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 11:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16825 to look at the new patch set (#7). Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables This patch adds support to INSERT INTO identity-partitioned Iceberg tables. Identity-partitioned Iceberg tables are similar to regular partitioned tables, they are even stored in the same directory structure. The difference is that the data files still store the partitioning columns. Partitioned Iceberg tables are stored as non-partitioned tables in the Hive Metastore (similarly to partitioned Kudu tables). However, the InsertStmt still generates the partition expressions for them. These partition expressions are used to shuffle and sort the input data so we don't end up writing too many files. The HdfsTableSink also uses the partition expressions to write the data files with the proper partition paths. Iceberg is able to parse the partition paths to generate the corresponding metadata for the partitions. This happens at the end in IcebergCatalogOpExecutor. Testing: * added planner test to verify shuffling and sorting * added negative tests for unsupported features like PARTITION clause and non-identity partition transforms * e2e tests with partitioned inserts TODO: * Current change includes some parts of IMPALA-10384 which needs to be removed once https://gerrit.cloudera.org/#/c/16850/ is merged Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/coordinator.cc M be/src/runtime/dml-exec-state.cc M be/src/service/client-request-state.cc M common/fbs/IcebergObjects.fbs M common/thrift/CatalogService.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test M tests/query_test/test_iceberg.py 25 files changed, 438 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/7 -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10343: increase control service queue mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16848 ) Change subject: IMPALA-10343: increase control_service_queue_mem_limit .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7fe93b5ce7eb6b63e48293ac287d98cc1d9e3fa Gerrit-Change-Number: 16848 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Dec 2020 11:51:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16825 to look at the new patch set (#5). Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables .. IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables This patch adds support to INSERT INTO identity-partitioned Iceberg tables. Identity-partitioned Iceberg tables are similar to regular partitioned tables, they are even stored in the same directory structure. The difference is that the data files still store the partitioning columns. Partitioned Iceberg tables are stored as non-partitioned tables in the Hive Metastore (similarly to partitioned Kudu tables). However, the InsertStmt still generates the partition expressions for them. These partition expressions are used to shuffle and sort the input data so we don't end up writing too many files. The HdfsTableSink also uses the partition expressions to write the data files with the proper partition paths. Iceberg is able to parse the partition paths to generate the corresponding metadata for the partitions. This happens at the end in IcebergCatalogOpExecutor. Testing: * added planner test to verify shuffling and sorting * added negative tests for unsupported features like PARTITION clause and non-identity partition transforms * e2e tests with partitioned inserts TODO: * Current change includes some parts of IMPALA-10384 which needs to be removed once https://gerrit.cloudera.org/#/c/16850/ is merged Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/coordinator.cc M be/src/runtime/dml-exec-state.cc M be/src/service/client-request-state.cc M common/fbs/IcebergObjects.fbs M common/thrift/CatalogService.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test M tests/query_test/test_iceberg.py 25 files changed, 438 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/5 -- To view, visit http://gerrit.cloudera.org:8080/16825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4 Gerrit-Change-Number: 16825 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 11 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 11:35:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6756/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 11 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 11:17:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
wangsheng has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. IMPALA-10361: Use field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. Currently, we use field id to resolve column for Iceberg tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid for Iceberg tables. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/query-options-test.cc M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/1-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/6-6-305c9b7a-f42d-4245-b806-dfa7a792593f-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/9-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/4-4-7a1a8e89-8aeb-4405-be64-76557432cf21-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-0.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-0.parquet A
[Impala-ASF-CR] IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6755/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 3 Gerrit-Owner: liuyao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 10:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7823/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 10:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7822/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 10:42:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16850 ) Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6754/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 10:33:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16850 to look at the new patch set (#2). Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. IMPALA-10384: Make partition names consistent between BE and FE In the BE we build partition names with the trailing char '/'. In the FE we build partition names without a trailing char. We should make this consistent because this causes some annoying string adjustments in the FE and can cause hidden bugs. This patch creates partition names without the trailing '/' both in the BE and the FE. This follows Hive's behavior that also prints partition names without the trailing '/'. Testing: * Ran exhaustive tests Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 --- M be/src/exec/hdfs-table-sink.cc M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionSdCompareTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M testdata/workloads/functional-query/queries/QueryTest/alter-table.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test M testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test M testdata/workloads/functional-query/queries/QueryTest/create-table.test M testdata/workloads/functional-query/queries/QueryTest/date-fileformat-support.test M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test M testdata/workloads/functional-query/queries/QueryTest/hdfs-caching.test M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/insert_null.test M testdata/workloads/functional-query/queries/QueryTest/insert_overwrite.test M testdata/workloads/functional-query/queries/QueryTest/insert_part_key.test M testdata/workloads/functional-query/queries/QueryTest/insert_permutation.test M testdata/workloads/functional-query/queries/QueryTest/load.test M testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test M testdata/workloads/tpcds-insert/queries/partitioned-insert.test 22 files changed, 740 insertions(+), 746 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16850/2 -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10384: Make partition names consistent between BE and FE
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16850 Change subject: IMPALA-10384: Make partition names consistent between BE and FE .. IMPALA-10384: Make partition names consistent between BE and FE In the BE we build partition names with the trailing char '/'. In the FE we build partition names without a trailing char. We should make this consistent because this causes some annoying string adjustments in the FE and can cause hidden bugs. This patch creates partition names without the trailing '/' both in the BE and the FE. This follows Hive's behavior that also prints partition names without the trailing '/'. Testing: * Ran exhaustive tests Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 --- M be/src/exec/hdfs-table-sink.cc M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionSdCompareTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M testdata/workloads/functional-query/queries/QueryTest/alter-table.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test M testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test M testdata/workloads/functional-query/queries/QueryTest/create-table.test M testdata/workloads/functional-query/queries/QueryTest/date-fileformat-support.test M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test M testdata/workloads/functional-query/queries/QueryTest/hdfs-caching.test M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/insert_null.test M testdata/workloads/functional-query/queries/QueryTest/insert_overwrite.test M testdata/workloads/functional-query/queries/QueryTest/insert_part_key.test M testdata/workloads/functional-query/queries/QueryTest/insert_permutation.test M testdata/workloads/functional-query/queries/QueryTest/load.test M testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test M testdata/workloads/tpcds-insert/queries/partitioned-insert.test 22 files changed, 740 insertions(+), 744 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16850/1 -- To view, visit http://gerrit.cloudera.org:8080/16850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7e40111e2d1148aeb01ebc985bbb15db7d6a6012 Gerrit-Change-Number: 16850 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7821/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 3 Gerrit-Owner: liuyao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Dec 2020 09:34:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6752/ -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 10 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 09:33:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate
liuyao has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate .. IMPALA-10377 Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate AggregationNode 1.The memory occupied by hash table's own data structure is not considered. Hash table inserts a new value, which will add a bucket. The size of a bucket is 16 bytes. 2.When estimating the NDV of merge aggregation, if there are multiple grouping exprs, it may be divided by the number of Fragment Instances several times, and it should be divided only once. 3.When estimating the NDV of merge aggregation, and there are multiple grouping exprs, the estimated memory is much smaller than the actual use. 4.If there is no grouping exprs, the estimated memory is much larger than the actual use. 5.If the NDV of grouping exprs is very small, the estimated memory is much larger than the actual use. SortNode 1.Estimate the memory usage of external sort. the estimated memory is much smaller than the actual use. HashJoinNode 1.The memory occupied by hash table's own data structure is not considered.Hash Table will keep duplicate data, so the size of DuplicateNode should be considered. 2.Hash table will create multiple buckets in advance. The size of these buckets should be considered. KuduScanNode 1.Estimate memory by scanning all columns,the estimated memory is much larger than the actual use. Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test M testdata/workloads/functional-planner/queries/PlannerTest/aggregation.test M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns-mt-dop.test M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test M testdata/workloads/functional-planner/queries/PlannerTest/card-inner-join.test M testdata/workloads/functional-planner/queries/PlannerTest/card-multi-join.test M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test M testdata/workloads/functional-planner/queries/PlannerTest/complex-types-file-formats.test M testdata/workloads/functional-planner/queries/PlannerTest/conjunct-ordering.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test M testdata/workloads/functional-planner/queries/PlannerTest/ddl.test M testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-broadcast.test M testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle.test M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/disable-preaggregations.test M testdata/workloads/functional-planner/queries/PlannerTest/distinct-estimate.test M testdata/workloads/functional-planner/queries/PlannerTest/distinct.test M testdata/workloads/functional-planner/queries/PlannerTest/empty.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test M testdata/workloads/functional-planner/queries/PlannerTest/insert-hdfs-writer-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/insert-sort-by-zorder.test M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-delete.test M
[Impala-ASF-CR] IMPALA-10386: Don't allow PARTITION BY SPEC for non-Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16846 ) Change subject: IMPALA-10386: Don't allow PARTITION BY SPEC for non-Iceberg tables .. Patch Set 1: Code-Review+1 This check is useful, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/16846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b3ec3e84476614cb11e801b6d89d84eb384dd43 Gerrit-Change-Number: 16846 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 09:02:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10360: Allow simple limit to be treated as sampling hint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16792 ) Change subject: IMPALA-10360: Allow simple limit to be treated as sampling hint .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6753/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife05a5343c913006f7659949b327b63d3f10c04b Gerrit-Change-Number: 16792 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 10 Dec 2020 08:52:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 10 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 08:28:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6752/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 10 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 08:12:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7819/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 10 Dec 2020 08:10:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables
wangsheng has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables .. IMPALA-10361: Use field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. Currently, we use field id to resolve column for Iceberg tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid for Iceberg tables. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/query-options-test.cc M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README M testdata/datasets/functional/functional_schema_template.sql M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/query_test/test_scanners.py 29 files changed, 453 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/10 -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 10 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command fails
Gabor Kaszab has abandoned this change. ( http://gerrit.cloudera.org:8080/16322 ) Change subject: IMPALA-10066: Fix test_cancellation_mid_command fails .. Abandoned There is another patch to address this: 16847 -- To view, visit http://gerrit.cloudera.org:8080/16322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ib80706d52a85d2c19b13fbbe5695934658c0bf7e Gerrit-Change-Number: 16322 Gerrit-PatchSet: 7 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
