[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 ) Change subject: IMPALA-11629: Support for huawei OBS FileSystem .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 12 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Feb 2023 02:53:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 ) Change subject: IMPALA-11629: Support for huawei OBS FileSystem .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9038/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 12 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Feb 2023 02:53:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem
lipeng...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 ) Change subject: IMPALA-11629: Support for huawei OBS FileSystem .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 11 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Feb 2023 02:52:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11903: Ozone emits NONE when not erasure-coded
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19482 ) Change subject: IMPALA-11903: Ozone emits NONE when not erasure-coded .. IMPALA-11903: Ozone emits NONE when not erasure-coded Updates Ozone support for identifying erasure-coded files/tables to emit NONE if not erasure-coded rather than the Ratis replication factor (e.g. ONE, THREE, etc). Chose to do this for consistency as the output specifically identifies the Erasure Coding Policy. Testing: - ran E2E tests with Ozone with and without EC Change-Id: I1c3a34d4e108fed38b66f3dabefe867be5441b35 Reviewed-on: http://gerrit.cloudera.org:8080/19482 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1c3a34d4e108fed38b66f3dabefe867be5441b35 Gerrit-Change-Number: 19482 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11903: Ozone emits NONE when not erasure-coded
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19482 ) Change subject: IMPALA-11903: Ozone emits NONE when not erasure-coded .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3a34d4e108fed38b66f3dabefe867be5441b35 Gerrit-Change-Number: 19482 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 08 Feb 2023 22:52:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11903: Ozone emits NONE when not erasure-coded
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19482 ) Change subject: IMPALA-11903: Ozone emits NONE when not erasure-coded .. Patch Set 1: Code-Review+2 This makes sense to me -- To view, visit http://gerrit.cloudera.org:8080/19482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3a34d4e108fed38b66f3dabefe867be5441b35 Gerrit-Change-Number: 19482 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 08 Feb 2023 20:45:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19484 ) Change subject: IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12342/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19484 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574 Gerrit-Change-Number: 19484 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 08 Feb 2023 20:36:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19484 Change subject: IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events .. IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events Added a map variable to store the processed reload event time for the corresponding table/partition. This will be updated frequently based on refresh or invalidate commands. This map can be used in the event processor to decide whether to process or skip the reload event by comparing the event time of the event with the latest event time from the map variable. Testing: Couldn't test this feature in an end-to-end test because the events would be filtered as self events. Tested it manually by removing the self event logic. Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 3 files changed, 97 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/19484/1 -- To view, visit http://gerrit.cloudera.org:8080/19484 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574 Gerrit-Change-Number: 19484 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11626: Handle COMMIT COMPACTION EVENT from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19155 ) Change subject: IMPALA-11626: Handle COMMIT_COMPACTION_EVENT from HMS .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12341/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19155 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I464faedb4e3bbcd417bab2e3cb0d57e339d42605 Gerrit-Change-Number: 19155 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 08 Feb 2023 20:10:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11626: Handle COMMIT COMPACTION EVENT from HMS
Hello Quanlong Huang, Daniel Becker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19155 to look at the new patch set (#9). Change subject: IMPALA-11626: Handle COMMIT_COMPACTION_EVENT from HMS .. IMPALA-11626: Handle COMMIT_COMPACTION_EVENT from HMS Since HIVE-24329 HMS emits an event when a compaction is committed, but Impala ignores it. Handling it would allow automatic refreshing of file metadata after commit compactions. Testing: Added an end-to-end test that tests the processiing of commit compaction event that was triggered in HMS. Also added an edge case where event processor would handle the condition of the partition being missed/deleted. Change-Id: I464faedb4e3bbcd417bab2e3cb0d57e339d42605 --- M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/resources/hive-site.xml.py M tests/custom_cluster/test_events_custom_configs.py 6 files changed, 196 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/19155/9 -- To view, visit http://gerrit.cloudera.org:8080/19155 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I464faedb4e3bbcd417bab2e3cb0d57e339d42605 Gerrit-Change-Number: 19155 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 18:00:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. IMPALA-11834: Error reporting addendum This commit is an addendum that addresses some review comments that arrived while the base change was merged. Code cleanup and improved error reporting. Testing: - Tested the code paths manually Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Reviewed-on: http://gerrit.cloudera.org:8080/19481 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/client-request-state.cc 1 file changed, 23 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11903: Ozone emits NONE when not erasure-coded
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19482 ) Change subject: IMPALA-11903: Ozone emits NONE when not erasure-coded .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9037/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3a34d4e108fed38b66f3dabefe867be5441b35 Gerrit-Change-Number: 19482 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 08 Feb 2023 17:41:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11892: Restore pkg resources with Python 2
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19467 ) Change subject: IMPALA-11892: Restore pkg_resources with Python 2 .. IMPALA-11892: Restore pkg_resources with Python 2 Impala's shell tarball used to include a copy of pkg_resources.py (from setuptools); due to the Python version we use for packaging, all modules with native libraries use pkg_resources to load the library. It was removed in IMPALA-9718 because Impala's copy of pkg_resources didn't work with Python 3. Some platforms - RHEL 7 for Python 2, Ubuntu/Debian - don't install setuptools by default as part of the python package, which causes impala-shell to error with "ImportError: No module named pkg_resources". Restores Impala's copy of pkg_resources.py to PYTHONPATH when running impala-shell under Python 2. Omits it for Python 3 so we use updated setuptools when available. python-setuptools will still be a manual requirement with Python 3. Testing - manually confirmed impala-shell starts in Ubuntu 20.04 docker container after 'apt install python' (omits setuptools). - manually confirmed impala-shell starts in Ubuntu 20.04 docker container after 'apt install python3-setuptools' (includes python). Change-Id: I78c05bce75ecc68de2296b1c2e57cd3c17c3cb0a Reviewed-on: http://gerrit.cloudera.org:8080/19467 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M shell/impala-shell M shell/make_shell_tarball.sh A shell/pkg_resources.py 3 files changed, 2,710 insertions(+), 2 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I78c05bce75ecc68de2296b1c2e57cd3c17c3cb0a Gerrit-Change-Number: 19467 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11859: Add bytes-read-encrypted metric
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19461 ) Change subject: IMPALA-11859: Add bytes-read-encrypted metric .. IMPALA-11859: Add bytes-read-encrypted metric Adds a metric bytes-read-encrypted to track encrypted reads. Testing: - ran test_io_metrics.py with Ozone (encrypts by default) - ran test_io_metrics.py with HDFS (no encryption) Change-Id: I9dbc194a4bc31cb0e01545fb6032a0853db60f34 Reviewed-on: http://gerrit.cloudera.org:8080/19461 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/scheduling/scheduler.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/fbs/CatalogObjects.fbs M common/protobuf/planner.proto M common/thrift/PlanNodes.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java D fe/src/main/java/org/apache/impala/compat/HdfsShim.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/query_test/test_io_metrics.py 20 files changed, 76 insertions(+), 46 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9dbc194a4bc31cb0e01545fb6032a0853db60f34 Gerrit-Change-Number: 19461 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11883: Calculate erasure-coded bytes read directly
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19460 ) Change subject: IMPALA-11883: Calculate erasure-coded bytes read directly .. IMPALA-11883: Calculate erasure-coded bytes read directly Calculate the metric erasure-coded-bytes-read directly from HDFS reads rather than through hdfsFileGetReadStatistics. This allows us to use it for other filesystem implementations (Ozone). Also renumbers is_erasure_coded in THdfsFileSplit to 8, where it was originally before it was removed by IMPALA-9485 (and never replaced). Testing: - ran updated test_io_metrics.py with Ozone, with and without EC - ran updated test_io_metrics.py with HDFS, with and without EC Change-Id: Ide0fc806590b2328df8068a9a54645d1d1fb137c Reviewed-on: http://gerrit.cloudera.org:8080/19460 Reviewed-by: Joe McDonnell Tested-by: Michael Smith --- M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/request-context.h M common/thrift/PlanNodes.thrift M tests/query_test/test_io_metrics.py 4 files changed, 9 insertions(+), 9 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Michael Smith: Verified -- To view, visit http://gerrit.cloudera.org:8080/19460 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ide0fc806590b2328df8068a9a54645d1d1fb137c Gerrit-Change-Number: 19460 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11883: Calculate erasure-coded bytes read directly
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19460 ) Change subject: IMPALA-11883: Calculate erasure-coded bytes read directly .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19460 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide0fc806590b2328df8068a9a54645d1d1fb137c Gerrit-Change-Number: 19460 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 08 Feb 2023 16:18:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 11 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 08 Feb 2023 16:15:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage
..
Patch Set 44:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG@16
PS44, Line 16: blocking-operator nature
May need to explain why blocking operators are considered in the context of DoP
adjustment.
Conceptually, blocking operators can be divided into multiple tiers with those
in the 1st are close to the leaf nodes and can run right away. Those in the 2nd
tier can run when all their dependent children (including the blocking
operators in 1st tier) can provide data. I did not see a linkage to increase
DoP for blocking operators.
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG@19
PS44, Line 19: explained
nit found
http://gerrit.cloudera.org:8080/#/c/19033/37/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:
http://gerrit.cloudera.org:8080/#/c/19033/37/be/src/util/backend-gflag-util.cc@201
PS37, Line 201: 0.5
> My intention is to associate the scaling flag to CPU requirement of the que
The scaling factor as defined is less intuitive, since one has to inverse it to
understand its semantics.
I think you can define the true scaling factor to be = 1 / .
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@147
PS37, Line 147: have an adjusted number of instance based on
: // Processin
nit. A positive value implies the instance count has been adjusted.
It is also nice to provide an example here.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@864
PS37, Line 864: elism, numNode
Better renamed as getMaxParallelismByTotalWorkSize().
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@883
PS37, Line 883:
: protected boolean hasAdjustedInstanceCount() { return
adjustedInstanceCount_ > 0; }
:
: protected void setFixedInstanceCount(int count) {
: isFixedParallelism_ = true;
: setAdjustedInstanceCount(count);
: }
Repeated use from line 869. Can be refactored.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@957
PS37, Line 957: processingCosts_.get(index).getNumInstanceExpected());
Should add a comment.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@972
PS37, Line 972:
: // Compute exchanging child parallelism first.
I wonder if the computation can be improved here e.g. by the size of the work.
Making it to the max # of nodes can overuse the system resource.
In general, I wonder if this logic tries to fix some bugs in DoP computation.
Adjusting DoP specifically for plans with blocking operators seems odd. See my
comment to the commit message.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@1098
PS37, Line 1098: if (hasBlockingNode()) {
add a comment should be helpful.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@1113
PS37, Line 1113:
does not sound right.
http://gerrit.cloudera.org:8080/#/c/19033/44/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test
File
testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test:
http://gerrit.cloudera.org:8080/#/c/19033/44/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test@75
PS44, Line 75: cardinality=3.04K
Is it possible to show the new processing cost here too? It will be wonderful.
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 44
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 08 Feb 2023 15:54:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
eubn...@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 1: Hello Quanlong Huang. Thanks for reply. I'll do it soon. -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 Feb 2023 15:16:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: [WIP] IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12340/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 15:06:42 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19483 )
Change subject: [WIP] IMPALA-11908: Parser change for Iceberg metadata querying
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19483/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/19483/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@414
PS1, Line 414: for (Column col :
IcebergSchemaConverter.convertToImpalaSchema(getIcebergSchema())) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19483/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java
File
fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java:
http://gerrit.cloudera.org:8080/#/c/19483/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java@55
PS1, Line 55: for (Column col :
IcebergSchemaConverter.convertToImpalaSchema(metadataTableSchema)) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19483/1/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19483/1/tests/query_test/test_iceberg.py@1006
PS1, Line 1006:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/19483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b
Gerrit-Change-Number: 19483
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Noemi Pap-Takacs
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 08 Feb 2023 14:46:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19483 Change subject: [WIP] IMPALA-11908: Parser change for Iceberg metadata querying .. [WIP] IMPALA-11908: Parser change for Iceberg metadata querying This commit is WIP, the design decisions have been made, some cleanup is yet to be done. This change extends parsing table references with Iceberg metadata tables. The TableName class has been extended with an extra vTbl field which is filled when a virtual table reference is suspected. This additional field helps to keep the real table in the statment table cache next to the virtual table. Which should be loaded so Iceberg metadat tables can be created. Iceberg provides a rich API to query metadata, these Iceberg API tables are accessible through the MetadataTableUtils class. Using these table schemas it is possible to create an Impala table that can be queried later on. Querying a metadata table at this point is expected to throw a NotImplementedExpception. Testing: - Added E2E test to test this feature. Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java A fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 13 files changed, 304 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19483/1 -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 2: Code-Review+2 Thanks. -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:50:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9036/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:51:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc@885 PS2, Line 885: errno > Optiional: it could be more informative if, before the errno value we wrote The strerror() call will translate the error number to an error message. -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:48:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc@885 PS2, Line 885: errno > That was actually unnecessary in L881, because the query_status is supposed Optiional: it could be more informative if, before the errno value we wrote something like "error number". -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:43:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 2: Code-Review+1 LGTM -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 12:35:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19471 ) Change subject: IMPALA-11081: Fix incorrect results in partition key scan .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12339/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 Gerrit-Change-Number: 19471 Gerrit-PatchSet: 12 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Wed, 08 Feb 2023 12:15:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 11: (2 comments) > Patch Set 9: > > Please provide more context on the actual queries and table sizes you are > targeting along with some performance measurements with and without this > optimization. Hi Kurt, take the 'tpcds. inventory' table as an example: the performance measurements result: https://issues.apache.org/jira/secure/attachment/13055252/performance_compare.txt http://gerrit.cloudera.org:8080/#/c/18705/9/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/18705/9/common/thrift/Query.thrift@602 PS9, Line 602: 149: optional bool trust_pk_fk_constraints = false > I think that this should be false by default, otherwise this is a potential Done in patch 11 http://gerrit.cloudera.org:8080/#/c/18705/6/testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test File testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test: http://gerrit.cloudera.org:8080/#/c/18705/6/testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test@1 PS6, Line 1: # Test Case Summary: > Can you add some simple EE tests too? e.g. on of the planner tests in https Done in patch 11. -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 11 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 08 Feb 2023 12:08:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19471 ) Change subject: IMPALA-11081: Fix incorrect results in partition key scan .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12338/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 Gerrit-Change-Number: 19471 Gerrit-PatchSet: 11 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Wed, 08 Feb 2023 12:05:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Yifan Zhang has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19471 ) Change subject: IMPALA-11081: Fix incorrect results in partition key scan .. IMPALA-11081: Fix incorrect results in partition key scan This patch fixes incorrect results caused by short-circuit partition key scan in the case where a Parquet/ORC file contains multiple blocks. IMPALA-8834 introduced the optimization that generating only one scan range that corresponding to the first block per file, backends only read footers for Parquet/ORC files, which leads to incorrect results if the first block doesn't include a file footer. This bug is fixed by returning a scan range corresponding to the last block for Parquet/ORC files to make sure it contains a file footer. Testing: - Added e2e tests to verify the fix. Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 --- M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/common/test_dimensions.py M tests/metadata/test_partition_metadata.py M tests/query_test/test_queries.py 4 files changed, 70 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/19471/12 -- To view, visit http://gerrit.cloudera.org:8080/19471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 Gerrit-Change-Number: 19471 Gerrit-PatchSet: 12 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19471 ) Change subject: IMPALA-11081: Fix incorrect results in partition key scan .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/19471/11/tests/common/test_dimensions.py File tests/common/test_dimensions.py: http://gerrit.cloudera.org:8080/#/c/19471/11/tests/common/test_dimensions.py@43 PS11, Line 43: } flake8: E123 closing bracket does not match indentation of opening bracket's line http://gerrit.cloudera.org:8080/#/c/19471/11/tests/query_test/test_queries.py File tests/query_test/test_queries.py: http://gerrit.cloudera.org:8080/#/c/19471/11/tests/query_test/test_queries.py@367 PS11, Line 367: c flake8: F841 local variable 'compression_codec' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/19471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 Gerrit-Change-Number: 19471 Gerrit-PatchSet: 11 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Wed, 08 Feb 2023 11:46:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Yifan Zhang has posted comments on this change. (
http://gerrit.cloudera.org:8080/19471 )
Change subject: IMPALA-11081: Fix incorrect results in partition key scan
..
Patch Set 11:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@347
PS10, Line 347: src_tbl_name = 'functional' + db_suffix + '.alltypes'
: src_tbl_loc = self._get_table_location(src_tbl_name, vector)
: source_file = src_tbl_loc + '/year=2010/month=12/*'
: tbl_loc =
get_fs_path("/test-warehouse/%s.db/alltypes_multiblocks"
: % (unique_database))
> I'm not clear why adding '_def' in the suffix. If you want the db_suffix of
Done
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@353
PS10, Line 353:
: check_call(['hdfs', 'dfs', '-mkdir', '-p', file_path
> I also see this in tests/metadata/test_partition_metadata.py. I think it's
Done
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@358
PS10, Line 358:
: # set block size to 1024 so the target file occupies
multiple blocks
: check_call(['hdfs', 'dfs', '-Ddfs.block.size=1024', '-cp',
'-f', '-d',
: source_file, file_path])
: self.client.execute("alter table %s.alltypes_multiblocks
recover partitions"
: % (unique_database))
:
> I think we can get the source table location by the _get_table_location() u
Done
--
To view, visit http://gerrit.cloudera.org:8080/19471
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1
Gerrit-Change-Number: 19471
Gerrit-PatchSet: 11
Gerrit-Owner: Yifan Zhang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Yifan Zhang
Gerrit-Comment-Date: Wed, 08 Feb 2023 11:46:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Yifan Zhang has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19471 ) Change subject: IMPALA-11081: Fix incorrect results in partition key scan .. IMPALA-11081: Fix incorrect results in partition key scan This patch fixes incorrect results caused by short-circuit partition key scan in the case where a Parquet/ORC file contains multiple blocks. IMPALA-8834 introduced the optimization that generating only one scan range that corresponding to the first block per file, backends only read footers for Parquet/ORC files, which leads to incorrect results if the first block doesn't include a file footer. This bug is fixed by returning a scan range corresponding to the last block for Parquet/ORC files to make sure it contains a file footer. Testing: - Added e2e tests to verify the fix. Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 --- M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/common/test_dimensions.py M tests/metadata/test_partition_metadata.py M tests/query_test/test_queries.py 4 files changed, 71 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/19471/11 -- To view, visit http://gerrit.cloudera.org:8080/19471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1 Gerrit-Change-Number: 19471 Gerrit-PatchSet: 11 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12337/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 11 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 08 Feb 2023 11:15:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9035/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 11 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 08 Feb 2023 11:05:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Xiang Yang has uploaded a new patch set (#11) to the change originally created by Jian Zhang. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. IMPALA-11417: Support outer join elimination optimization When two tables are outer joined but only fields from the outer side table are used and the join key of the inner side table is guaranteed to be unique, the query can be simplified to only scan the outer table: drop table if exists t; drop table if exists s; create table t(sid bigint, value bigint); create table s(id bigint, value bigint, primary key(id)); -- the test SQL: select t.* from t left join s on t.sid = s.id; The above query can be simplified to: select t.* from t; This optimization utilizes the primary key constraint when creating join nodes, eliminates the inner side when the join key on inner side is the primary key and only the slots from the outer side are used by the parent. Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Signed-off-by: Jian Zhang --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test A testdata/workloads/tpcds/queries/outer-join-elimination.test M tests/query_test/test_join_queries.py 11 files changed, 535 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/18705/11 -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 11 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12336/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 10 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 08 Feb 2023 10:30:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/18705 )
Change subject: IMPALA-11417: Support outer join elimination optimization
..
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/18705/10/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/18705/10/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2071
PS10, Line 2071: && canEliminateInnerPlan(analyzer, inner, innerRef,
eqJoinConjuncts, otherJoinConjuncts)) {
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/18705/10/tests/query_test/test_join_queries.py
File tests/query_test/test_join_queries.py:
http://gerrit.cloudera.org:8080/#/c/18705/10/tests/query_test/test_join_queries.py@89
PS10, Line 89: d
flake8: F811 redefinition of unused 'test_outer_joins' from line 83
http://gerrit.cloudera.org:8080/#/c/18705/10/tests/query_test/test_join_queries.py@121
PS10, Line 121: class TestJoinElimination(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/18705/10/tests/query_test/test_join_queries.py@131
PS10, Line 131: \
flake8: E502 the backslash is redundant between brackets
--
To view, visit http://gerrit.cloudera.org:8080/18705
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece
Gerrit-Change-Number: 18705
Gerrit-PatchSet: 10
Gerrit-Owner: Jian Zhang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jian Zhang
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiang Yang
Gerrit-Comment-Date: Wed, 08 Feb 2023 10:08:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Xiang Yang has uploaded a new patch set (#10) to the change originally created by Jian Zhang. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. IMPALA-11417: Support outer join elimination optimization When two tables are outer joined but only fields from the outer side table are used and the join key of the inner side table is guaranteed to be unique, the query can be simplified to only scan the outer table: drop table if exists t; drop table if exists s; create table t(sid bigint, value bigint); create table s(id bigint, value bigint, primary key(id)); -- the test SQL: select t.* from t left join s on t.sid = s.id; The above query can be simplified to: select t.* from t; This optimization utilizes the primary key constraint when creating join nodes, eliminates the inner side when the join key on inner side is the primary key and only the slots from the outer side are used by the parent. Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Signed-off-by: Jian Zhang --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test A testdata/workloads/tpcds/queries/outer-join-elimination.test M tests/query_test/test_join_queries.py 11 files changed, 539 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/18705/10 -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 10 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-11823: Add more items to Impala web UI queries page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19417 ) Change subject: IMPALA-11823: Add more items to Impala web UI queries page .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19417 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19c75461a6405025fa433ae84d2c94d013fcaacb Gerrit-Change-Number: 19417 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Wed, 08 Feb 2023 08:24:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11081: Fix incorrect results in partition key scan
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/19471 )
Change subject: IMPALA-11081: Fix incorrect results in partition key scan
..
Patch Set 10:
(3 comments)
The fix looks good to me. Just have some suggestions in the test.
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@347
PS10, Line 347: db_suffix = file_format + '_' + compression_codec
: if file_format == 'parquet' and compression_codec == 'none':
: db_suffix = 'parquet'
: elif compression_codec == 'none':
: db_suffix = file_format + '_def'
I'm not clear why adding '_def' in the suffix. If you want the db_suffix of the
file format, you can use:
vector.get_value('table_format').db_suffix()
https://github.com/apache/impala/blob/32536ba25869ea621bdb82bb79a520a115e02cdd/tests/common/test_dimensions.py#L90-L98
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@353
PS10, Line 353: STORED_AS_ARGS = {'text': 'textfile', 'parquet': 'parquet',
'avro': 'avro',
: 'seq': 'sequencefile', 'orc': 'orc', 'rc': 'rcfile'}
I also see this in tests/metadata/test_partition_metadata.py. I think it's
useful for other tests. Let's move this to a common place so it can be shared.
E.g. move it to tests/common/test_dimensions.py with more items from the
complete map:
https://github.com/apache/impala/blob/1d05381b7b791bde5820572b3e7a4b2b5db1db73/testdata/bin/generate-schema-statements.py#L196-L207
http://gerrit.cloudera.org:8080/#/c/19471/10/tests/query_test/test_queries.py@358
PS10, Line 358: source_file =
get_fs_path("/test-warehouse/alltypes_%s/year=2010/month=12/*"
: % (db_suffix))
: if file_format == 'orc':
: # functional_orc_def.alltypes is a managed table.
: source_file = get_fs_path(
:
"/test-warehouse/managed/functional_%s.db/alltypes_%s/year=2010/month=12/*"
: % (db_suffix, db_suffix))
I think we can get the source table location by the _get_table_location() util
method:
src_tbl_name = 'functional' + db_suffix + '.alltypes'
src_tbl_loc = self._get_table_location(src_tbl_name, vector)
source_file = src_tbl_loc + '/year=2010/month=12'
--
To view, visit http://gerrit.cloudera.org:8080/19471
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17331ed6c26a747e0509dcbaf427cd52808943b1
Gerrit-Change-Number: 19471
Gerrit-PatchSet: 10
Gerrit-Owner: Yifan Zhang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Yifan Zhang
Gerrit-Comment-Date: Wed, 08 Feb 2023 08:19:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11823: Add more items to Impala web UI queries page
18770832...@163.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/19417 ) Change subject: IMPALA-11823: Add more items to Impala web UI queries page .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19417/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/19417/2/be/src/runtime/coordinator.cc@1429 PS2, Line 1429: UpdatePeakMemUsageAndBytesRead > nit: Maybe we can use the existing Coordinator::ComputeQueryResourceUtiliza Done -- To view, visit http://gerrit.cloudera.org:8080/19417 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19c75461a6405025fa433ae84d2c94d013fcaacb Gerrit-Change-Number: 19417 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Wed, 08 Feb 2023 08:15:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11834: Error reporting addendum
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19481 ) Change subject: IMPALA-11834: Error reporting addendum .. Patch Set 2: (1 comment) Thank you for the review Daniel! http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19481/2/be/src/service/client-request-state.cc@885 PS2, Line 885: errno > Before this change we added "query_status.msg().msg()" to the status error That was actually unnecessary in L881, because the query_status is supposed to be OK on this code path. On the other code path however it is useful to add the query_status and it is appended to revert_err. Changed to GetDetail() because this method is better to get the error message from the Status object. -- To view, visit http://gerrit.cloudera.org:8080/19481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece31fa19b7448095462671acb2f02e8c5a405ae Gerrit-Change-Number: 19481 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Feb 2023 08:11:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11823: Add more items to Impala web UI queries page
Hello Quanlong Huang, Kurt Deschler, Yifan Zhang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19417 to look at the new patch set (#6). Change subject: IMPALA-11823: Add more items to Impala web UI queries page .. IMPALA-11823: Add more items to Impala web UI queries page When operating and maintaining an Impala cluster or analyzing historical query performance, it will be helpful if we show the memory consumed, the amount of data read, and other information of the query from the historical query page of the web UI. The current historical query page does not display these information, so we should count this information when the query is executed and display it on the web page. Change-Id: I19c75461a6405025fa433ae84d2c94d013fcaacb --- M be/src/runtime/coordinator.h M be/src/scheduling/admission-control-client.h M be/src/scheduling/admission-control-service.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/local-admission-control-client.cc M be/src/scheduling/local-admission-control-client.h M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/remote-admission-control-client.h M be/src/scheduling/schedule-state.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/protobuf/admission_control_service.proto M www/queries.tmpl M www/query_detail_tabs.tmpl M www/query_plan.tmpl M www/query_summary.tmpl 23 files changed, 474 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/19417/6 -- To view, visit http://gerrit.cloudera.org:8080/19417 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I19c75461a6405025fa433ae84d2c94d013fcaacb Gerrit-Change-Number: 19417 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang
