[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15890/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 15 Apr 2024 14:01:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12990: Fix impala-shell handling of unset rows deleted
Hello Peter Rozsa, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21284 to look at the new patch set (#6). Change subject: IMPALA-12990: Fix impala-shell handling of unset rows_deleted .. IMPALA-12990: Fix impala-shell handling of unset rows_deleted The issue occurred in Python 3 when 0 rows were deleted from Iceberg. It could also happen in other DMLs with older Impala servers where TDmlResult.rows_deleted was not set. See the Jira for details of the error. Testing: Extended shell tests for Kudu DML reporting to also cover Iceberg. Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 --- M shell/impala_client.py M shell/impala_shell.py M tests/custom_cluster/test_hs2_fault_injection.py M tests/shell/test_shell_commandline.py 4 files changed, 97 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/21284/6 -- To view, visit http://gerrit.cloudera.org:8080/21284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 Gerrit-Change-Number: 21284 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/21257/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21257/11//COMMIT_MSG@19 PS11, Line 19: rather : than sum of it (48) > Can this get higher than the amount of slots per executor? Based on IMPALA- It can be, if individual executor is configured with mismatch --admission_control_slots value than what is expressed in llama-site.xml of coordinator (field name like "impala.admission-control.max-query-cpu-core-per-node-limit.root.large", for root.large executor group). This patch assume correct configuration between llama-site.xml of coordinator and --admission_control_slots value of each backend executors. http://gerrit.cloudera.org:8080/#/c/21257/11//COMMIT_MSG@28 PS11, Line 28: which will be a closer resemblance of maximum : parallel execution of fragment instances. > Does PLANNER_CPU_ASK always calculate a greater or equal number of slots th It should be most of the time, except few corner cases such as dedicated coordinator setup. In that setup, there might be no dominant fragment assigned to coordinator at all. This is handled by L1027 at scheduler.cc, patch set 11. Now that I think again, I might miss other corner cases where dominant_instance_count < be_max_instances, I just have not found such example. I think be_max_instances should win in that case. Will fix it in next patch set. -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 16:09:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Hello Kurt Deschler, Abhishek Rawat, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21257 to look at the new patch set (#13). Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. IMPALA-12980: Translate CpuAsk into admission control slots Impala has a concept of "admission control slots" - the amount of parallelism that should be allowed on an Impala daemon. This defaults to the number of processors per executor and can be overridden with -–admission_control_slots flag. Admission control slot accounting is described in IMPALA-8998. It compute 'slots_to_use' for each backend based on the max number of instances of any fragment on that backend. This can lead to underestimation, because multiple non-blocking fragments with the same number of instance counts, say 4 non-blocking fragments each with 12 instances, only request the max instance (12) admission slots rather than sum of it (48), making each of 12 cores oversubscribed by 4x. When COMPUTE_PROCESSING_COST is enabled, Planner will generate a CpuAsk number that represents the cpu requirement of that query over a particular executor group set. This number is an estimation of the largest number of query fragment instances that can run in parallel without waiting, given by the blocking operator analysis. Therefore, the fragment trace that sums into that CpuAsk number can be translated into 'slots_to_use' as well, which will be a closer resemblance of maximum parallel execution of fragment instances. This patch adds a new query option called SLOT_COUNT_STRATEGY to control which admission control slot accounting to use. There are two possible values: - LARGEST_FRAGMENT, which is the original algorithm from IMPALA-8998. This is still the default value for the SLOT_COUNT_STRATEGY option. - PLANNER_CPU_ASK, which will follow the fragment trace that contributes towards CpuAsk number. This strategy will schedule more admission control slots than the LARGEST_FRAGMENT strategy. To do the PLANNER_CPU_ASK strategy, the Planner will mark fragments that contribute to CpuAsk as dominant fragments. It also passes max_slot_per_executor information that it knows about the executor group set to the scheduler. AvgAdmissionSlotsPerExecutor counter is added to describe what Planner thinks the average 'slots_to_use' per backend will be, which follows this formula: AvgAdmissionSlotsPerExecutor = ceil(CpuAsk / num_executors) Actual 'slots_to_use' in each backend may differ than AvgAdmissionSlotsPerExecutor, depending on what is scheduled on that backend. 'slots_to_use' will be shown as 'AdmissionSlots' counter under each executor profile node. Testing: - Update test_executors.py with AvgAdmissionSlotsPerExecutor assertion. - Pass test_tpcds_queries.py::TestTpcdsQueryWithProcessingCost. - Add EE test test_processing_cost.py. Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Planner.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/common/Id.java M fe/src/main/java/org/apache/impala/planner/CoreCount.java M fe/src/main/java/org/apache/impala/planner/CostingSegment.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test M tests/custom_cluster/test_executor_groups.py A tests/query_test/test_processing_cost.py M tests/query_test/test_tpcds_queries.py 19 files changed, 510 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/21257/13 -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 13 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#10). Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. IMPALA-12933: Avoid fetching unneccessary events of unwanted types There are several places that catalogd will fetch all events of a specific type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. To avoid bringing too much computation overhead to HMS's underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag, common_hms_event_types, is added to specify the common HMS event types. Once HIVE-28146 is resolved, we can set the wanted types directly in the HMS RPC. This approach can be simplified. UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most common unused events for Impala. They are also added to the default skip list. A new flag, common_hms_event_types, is added to configure this list. This patch also fixes an issue that events of the non-default catalog are not filtered out. In a local perf test, I generated 100K RELOAD events after creating a table in Hive. Then use the table in Impala to trigger metadata loading on it which will fetch the latest CREATE_TABLE event by polling all events after the last known CREATE_TABLE event. Before this patch, fetching the events takes 1s779ms. Now it takes only 395.377ms. Note that in prod env, the event messages are usually larger, we could have a larger speedup. Tests: - Added an FE test - Ran CORE tests Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 15 files changed, 285 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/10 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Yida Wu has posted comments on this change. (
http://gerrit.cloudera.org:8080/21294 )
Change subject: IMPALA-12874: Identify active and standby catalog and
statestore the web debug endpoint
..
Patch Set 2:
(2 comments)
Thanks Wenzhe for reviewing.
http://gerrit.cloudera.org:8080/#/c/21294/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21294/1//COMMIT_MSG@15
PS1, Line 15: the
: status as
> nit: status as active
Done
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc@283
PS1, Line 283: if (metric->GetValue()) {
: document->AddMember(
: "catalogd_active_status", "Active",
document->GetAllocator());
: } else {
: document->AddMember(
: "catalogd_active_status", "Standby",
document->GetAllocator());
: }
> nit: this can be simplified as
Tried, but the initializer of rapidjson::GenericValue seems not working this
way..
--
To view, visit http://gerrit.cloudera.org:8080/21294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2
Gerrit-Change-Number: 21294
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 15 Apr 2024 17:06:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21294 ) Change subject: IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint .. IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint This patch adds support to display the HA status of catalog and statestore on the root web page. The status will be presented as "Catalog Status: Active" or "Statestore Status: Standby" based on the values retrieved from the metrics catalogd-server.active-status and statestore.active-status. If the catalog or statestore is standalone, it will show the status as active, which is same as the metric. Tests: Ran core tests. Manually tests the web page, and verified the status display is correct. Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2 --- M be/src/common/daemon-env.h M be/src/util/default-path-handlers.cc M be/src/util/default-path-handlers.h M www/root.tmpl 4 files changed, 61 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/21294/2 -- To view, visit http://gerrit.cloudera.org:8080/21294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2 Gerrit-Change-Number: 21294 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Hello Kurt Deschler, Abhishek Rawat, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21257 to look at the new patch set (#12). Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. IMPALA-12980: Translate CpuAsk into admission control slots Impala has a concept of "admission control slots" - the amount of parallelism that should be allowed on an Impala daemon. This defaults to the number of processors per executor and can be overridden with -–admission_control_slots flag. Admission control slot accounting is described in IMPALA-8998. It compute 'slots_to_use' for each backend based on the max number of instances of any fragment on that backend. This can lead to underestimation, because multiple non-blocking fragments with the same number of instance counts, say 4 non-blocking fragments each with 12 instances, only request the max instance (12) admission slots rather than sum of it (48), making each of 12 cores oversubscribed by 4x. When COMPUTE_PROCESSING_COST is enabled, Planner will generate a CpuAsk number that represents the cpu requirement of that query over a particular executor group set. This number is an estimation of the largest number of query fragment instances that can run in parallel without waiting, given by the blocking operator analysis. Therefore, the fragment trace that sums into that CpuAsk number can be translated into 'slots_to_use' as well, which will be a closer resemblance of maximum parallel execution of fragment instances. This patch adds a new query option called SLOT_COUNT_STRATEGY to control which admission control slot accounting to use. There are two possible values: - LARGEST_FRAGMENT, which is the original algorithm from IMPALA-8998. This is still the default value for the SLOT_COUNT_STRATEGY option. - PLANNER_CPU_ASK, which will follow the fragment trace that contributes towards CpuAsk number. This strategy will schedule more admission control slots than the LARGEST_FRAGMENT strategy. To do the PLANNER_CPU_ASK strategy, the Planner will mark fragments that contribute to CpuAsk as dominant fragments. It also passes max_slot_per_executor information that it knows about the executor group set to the scheduler. AvgAdmissionSlotsPerExecutor counter is added to describe what Planner thinks the average 'slots_to_use' per backend will be, which follows this formula: AvgAdmissionSlotsPerExecutor = ceil(CpuAsk / num_executors) Actual 'slots_to_use' in each backend may differ than AvgAdmissionSlotsPerExecutor, depending on what is scheduled on that backend. 'slots_to_use' will be shown as 'AdmissionSlots' counter under each executor profile node. Testing: - Update test_executors.py with AvgAdmissionSlotsPerExecutor assertion. - Pass test_tpcds_queries.py::TestTpcdsQueryWithProcessingCost. Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Planner.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/common/Id.java M fe/src/main/java/org/apache/impala/planner/CoreCount.java M fe/src/main/java/org/apache/impala/planner/CostingSegment.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test M tests/custom_cluster/test_executor_groups.py A tests/query_test/test_processing_cost.py M tests/query_test/test_tpcds_queries.py 19 files changed, 507 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/21257/12 -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 12: (4 comments) http://gerrit.cloudera.org:8080/#/c/21257/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21257/11//COMMIT_MSG@28 PS11, Line 28: which will be a closer resemblance of maximum : parallel execution of fragment instances. > It should be most of the time, except few corner cases such as dedicated co Done http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test File testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test: http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test@2 PS11, Line 2: QUERY: TPCDS-Q1-CPC-PLANNER-CPU-ASK > The EXPLAIN plan should be equal to the one in workloads/functional-planner Done http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test@31 PS11, Line 31: order by c_customer_id > Will delete this RESULTS section. Done http://gerrit.cloudera.org:8080/#/c/21257/11/tests/query_test/test_processing_cost.py File tests/query_test/test_processing_cost.py: http://gerrit.cloudera.org:8080/#/c/21257/11/tests/query_test/test_processing_cost.py@42 PS11, Line 42: def test_admission_slots(self, vector): > Will do. Done -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 17:32:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#13). Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. IMPALA-12933: Avoid fetching unneccessary events of unwanted types There are several places where catalogd will fetch all events of a specific type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. To avoid bringing too much computation overhead to HMS's underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag, common_hms_event_types, is added to specify the common HMS event types. Once HIVE-28146 is resolved, we can set the wanted types directly in the HMS RPC. This approach can be simplified. UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most common unused events for Impala. They are also added to the default skip list. A new flag, common_hms_event_types, is added to configure this list. This patch also fixes an issue that events of the non-default catalog are not filtered out. In a local perf test, I generated 100K RELOAD events after creating a table in Hive. Then use the table in Impala to trigger metadata loading on it which will fetch the latest CREATE_TABLE event by polling all events after the last known CREATE_TABLE event. Before this patch, fetching the events takes 1s779ms. Now it takes only 395.377ms. Note that in prod env, the event messages are usually larger, we could have a larger speedup. Tests: - Added an FE test - Ran CORE tests Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 15 files changed, 286 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/13 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/21186 )
Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted
types
..
Patch Set 13:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/21186/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21186/9//COMMIT_MSG@9
PS9, Line 9: wher
> nit: "where"
Done
http://gerrit.cloudera.org:8080/#/c/21186/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/21186/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@361
PS10, Line 361: // CREATE_FUNCTION, DROP_FUNCTION, ADD_PRIMARYKEY,
ADD_FOREIGNKEY, ADD_UNIQUECONSTRAINT,
> line too long (136 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@281
PS9, Line 281:
:
Also added a flag for this.
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@284
PS9, Line 284:
> Can you mention this in the commit message?
Done
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@297
PS9, Line 297:* Wrapper around {@link
#getNextMetastoreEventsInBatches(CatalogServiceCatalog, long,
> Wouldn't it be better to get this list from a flag? I can imagine both case
Good point. Added a flag for this.
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@394
PS9, Line 394: result.size(), numFilteredEvents, toEventId -
eventId, eventId + 1);
> It would be nice to have a similar log when get all event types.
Done
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/21186/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2134
PS9, Line 2134: equals
> At other places equalsIgnoreCase is used
We use equals() on event types and equalsIgnoreCase() on catalog/db/table
names. I think it's ok to keep it. BTW, using equalsIgnoreCase() makes L2770
super long and hard to format..
--
To view, visit http://gerrit.cloudera.org:8080/21186
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9
Gerrit-Change-Number: 21186
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Mon, 15 Apr 2024 13:38:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]Hierarchical metastore event processing
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21031 )
Change subject: [WIP]Hierarchical metastore event processing
..
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21031/9/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java:
http://gerrit.cloudera.org:8080/#/c/21031/9/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java@269
PS9, Line 269: "[Performance] Non-Partitioned transactional table with
hierarchical processing: {}",
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/21031/9/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java@272
PS9, Line 272: "tblproperties ('transactional'='true',
'transactional_properties'='insert_only')");
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/21031/9/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java@289
PS9, Line 289: "tblproperties ('transactional'='true',
'transactional_properties'='insert_only')");
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 15 Apr 2024 14:32:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12990: Fix impala-shell handling of unset rows deleted
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21284 ) Change subject: IMPALA-12990: Fix impala-shell handling of unset rows_deleted .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21284/2/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/21284/2/shell/impala_client.py@980 PS2, Line 980: num_deleted_rows = 0 > Ah, I assumed the comment referred to num_deleted_rows but looks like it wa changed it back to returning None instead - it seemed more logical to me to be able to differentiate in the caller between 0 and not having a value. Fixed this in the caller side to handle None. + refactored the row count report formatting a bit to add less noise to _execute_stmt -- To view, visit http://gerrit.cloudera.org:8080/21284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 Gerrit-Change-Number: 21284 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 15 Apr 2024 14:49:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21301 ) Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10541/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Apr 2024 16:34:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21301 Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly If the Iceberg table has Avro delete files (e.g. by setting 'write.delete.format.default'='avro') then Impala won't be able to read the contents of the delete files properly. It is because the avro schema is not set properly for the virtual delete table. Testing: * added e2e tests with position delete files of all kinds Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 --- M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test M tests/query_test/test_iceberg.py 3 files changed, 143 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/21301/1 -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21301 ) Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15895/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Apr 2024 16:58:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15889/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 15 Apr 2024 13:53:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12963: Return parent PID when children spawned
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21278 ) Change subject: IMPALA-12963: Return parent PID when children spawned .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Gerrit-Change-Number: 21278 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Apr 2024 16:44:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12963: Return parent PID when children spawned
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21278 ) Change subject: IMPALA-12963: Return parent PID when children spawned .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Gerrit-Change-Number: 21278 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Apr 2024 16:54:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12963: Return parent PID when children spawned
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21278 ) Change subject: IMPALA-12963: Return parent PID when children spawned .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10542/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Gerrit-Change-Number: 21278 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Apr 2024 16:54:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#12). Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. IMPALA-12933: Avoid fetching unneccessary events of unwanted types There are several places where catalogd will fetch all events of a specific type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. To avoid bringing too much computation overhead to HMS's underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag, common_hms_event_types, is added to specify the common HMS event types. Once HIVE-28146 is resolved, we can set the wanted types directly in the HMS RPC. This approach can be simplified. UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most common unused events for Impala. They are also added to the default skip list. A new flag, common_hms_event_types, is added to configure this list. This patch also fixes an issue that events of the non-default catalog are not filtered out. In a local perf test, I generated 100K RELOAD events after creating a table in Hive. Then use the table in Impala to trigger metadata loading on it which will fetch the latest CREATE_TABLE event by polling all events after the last known CREATE_TABLE event. Before this patch, fetching the events takes 1s779ms. Now it takes only 395.377ms. Note that in prod env, the event messages are usually larger, we could have a larger speedup. Tests: - Added an FE test - Ran CORE tests Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 15 files changed, 286 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/12 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] [WIP]Hierarchical metastore event processing
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21031 to look at the new patch set (#9). Change subject: [WIP]Hierarchical metastore event processing .. [WIP]Hierarchical metastore event processing Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/TableWriteId.java A fe/src/main/java/org/apache/impala/catalog/events/DBBarrierEvent.java A fe/src/main/java/org/apache/impala/catalog/events/DBEventExecutor.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/catalog/CatalogTableWriteIdTest.java A fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorPerfTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 16 files changed, 1,453 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21031/9 -- To view, visit http://gerrit.cloudera.org:8080/21031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 Gerrit-Change-Number: 21031 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [WIP]Hierarchical metastore event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21031 ) Change subject: [WIP]Hierarchical metastore event processing .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15892/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 Gerrit-Change-Number: 21031 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Apr 2024 14:55:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list Binary fields in complex types are currently not supported at all for regular tables (an error is returned). For Iceberg metadata tables, IMPALA-12899 added a temporary workaround to allow queries that contain these fields to succeed by NULLing them out. This change adds support for displaying them with base64 encoding for both regular and Iceberg metadata tables. Complex types are displayed in JSON format, so simply inserting the bytes of the binary fields is not acceptable as it would produce invalid JSON. Base64 is a widely used encoding that allows representing arbitrary binary information using only a limited set of ASCII characters. This change also adds support for top level binary columns in Iceberg metadata tables. However, these are not base64 encoded but are returned in raw byte format - this is consistent with how top level binary columns from regular (non-metadata) tables are handled. Testing: - added test queries in iceberg-metadata-tables.test referencing both nested and top level binary fields; also updated existing queries - moved relevant tests (queries extracting binary fields from within complex types) from nested-types-scanner-basic.test to a new binary-in-complex-type.test file and also added a query that selects the containing complex types; this new test file is run from test_scanners.py::TestBinaryInComplexType::\ test_binary_in_complex_type - moved negative tests in AnalyzerTest.TestUnsupportedTypes() to AnalyzeStmtsTest.TestComplexTypesInSelectList() and converted them to positive tests (expecting success); a negative test already in AnalyzeStmtsTest.TestComplexTypesInSelectList() was also converted Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 --- M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M be/src/rpc/jni-thrift-util.h M be/src/runtime/complex-value-writer.inline.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/data/0-0-data-danielbecker_20240408174043_c3737eaf-db30-4b88-aafb-f23c0f3c1dd3-job_17125053806420_0002-1-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/64da0e56-efa3-4025-bef1-1047fdd9a2b0-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/snap-3079551887386250470-1-64da0e56-efa3-4025-bef1-1047fdd9a2b0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/binary-in-complex-type.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test M tests/query_test/test_scanners.py 26 files changed, 439 insertions(+), 154 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/21269/6 -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21269/6/be/src/util/jni-util.h File be/src/util/jni-util.h: http://gerrit.cloudera.org:8080/#/c/21269/6/be/src/util/jni-util.h@115 PS6, Line 115: /// is more restricted, see https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetPrimitiveArrayCritical_ReleasePrimitiveArrayCritical line too long (162 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 15 Apr 2024 15:15:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21294 ) Change subject: IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15896/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2 Gerrit-Change-Number: 21294 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Apr 2024 17:31:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21257/12/tests/query_test/test_processing_cost.py File tests/query_test/test_processing_cost.py: http://gerrit.cloudera.org:8080/#/c/21257/12/tests/query_test/test_processing_cost.py@22 PS12, Line 22: from tests.common.test_dimensions import ( flake8: F401 'tests.common.test_dimensions.add_mandatory_exec_option' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 17:31:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15891/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 15 Apr 2024 14:02:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21257 )
Change subject: IMPALA-12980: Translate CpuAsk into admission control slots
..
Patch Set 11:
(4 comments)
Thank you for the review so far, I will push a new patch set today.
http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test
File
testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test:
http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test@2
PS11, Line 2: QUERY
> It would be very nice to add an explain for the same query with the same qu
The EXPLAIN plan should be equal to the one in
workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-q01.test.
I will mention it as a comment.
The RUNTIME_PROFILE section in this test assert FrontendProfile section that
does not belong to the EXPLAIN part. Therefore, this is written as EE test and
not FE test.
http://gerrit.cloudera.org:8080/#/c/21257/11/testdata/workloads/functional-query/queries/QueryTest/processing_cost_admission_slots.test@31
PS11, Line 31: RESULTS
> Are the results actually important here?
Will delete this RESULTS section.
http://gerrit.cloudera.org:8080/#/c/21257/11/tests/custom_cluster/test_executor_groups.py
File tests/custom_cluster/test_executor_groups.py:
http://gerrit.cloudera.org:8080/#/c/21257/11/tests/custom_cluster/test_executor_groups.py@1245
PS11, Line 1245: # CoreCount={total=16
trace=F15:3+F01:1+F14:3+F03:1+F13:3+F05:1+F12:3+F07:1},
> This is very useful to help in understanding the patch, but it would be bet
I will mention the planner test for TPC-DS Q1 here.
http://gerrit.cloudera.org:8080/#/c/21257/11/tests/query_test/test_processing_cost.py
File tests/query_test/test_processing_cost.py:
http://gerrit.cloudera.org:8080/#/c/21257/11/tests/query_test/test_processing_cost.py@42
PS11, Line 42: add_mandatory_exec_option(cls, 'slot_count_strategy',
'planner_cpu_ask')
> IMO moving these to SET statements in the .test file would be clearer. It w
Will do.
--
To view, visit http://gerrit.cloudera.org:8080/21257
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2
Gerrit-Change-Number: 21257
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 15 Apr 2024 15:40:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21294 )
Change subject: IMPALA-12874: Identify active and standby catalog and
statestore the web debug endpoint
..
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/21294/1/be/src/util/default-path-handlers.cc@283
PS1, Line 283: if (metric->GetValue()) {
: document->AddMember(
: "catalogd_active_status", "Active",
document->GetAllocator());
: } else {
: document->AddMember(
: "catalogd_active_status", "Standby",
document->GetAllocator());
: }
> Tried, but the initializer of rapidjson::GenericValue seems not working thi
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2
Gerrit-Change-Number: 21294
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Mon, 15 Apr 2024 17:22:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#11). Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. IMPALA-12933: Avoid fetching unneccessary events of unwanted types There are several places where catalogd will fetch all events of a specific type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. To avoid bringing too much computation overhead to HMS's underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag, common_hms_event_types, is added to specify the common HMS event types. Once HIVE-28146 is resolved, we can set the wanted types directly in the HMS RPC. This approach can be simplified. UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most common unused events for Impala. They are also added to the default skip list. A new flag, common_hms_event_types, is added to configure this list. This patch also fixes an issue that events of the non-default catalog are not filtered out. In a local perf test, I generated 100K RELOAD events after creating a table in Hive. Then use the table in Impala to trigger metadata loading on it which will fetch the latest CREATE_TABLE event by polling all events after the last known CREATE_TABLE event. Before this patch, fetching the events takes 1s779ms. Now it takes only 395.377ms. Note that in prod env, the event messages are usually larger, we could have a larger speedup. Tests: - Added an FE test - Ran CORE tests Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 15 files changed, 285 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/11 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/15894/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 15 Apr 2024 15:38:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15897/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 17:52:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12990: Fix impala-shell handling of unset rows deleted
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21284 ) Change subject: IMPALA-12990: Fix impala-shell handling of unset rows_deleted .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15893/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 Gerrit-Change-Number: 21284 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 15 Apr 2024 15:06:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15898/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 13 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 18:27:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12998: Add SHOW METADATA TABLES to ignored DDL
Hello Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21290 to look at the new patch set (#2). Change subject: IMPALA-12998: Add SHOW_METADATA_TABLES to ignored DDL .. IMPALA-12998: Add SHOW_METADATA_TABLES to ignored DDL Adds SHOW_METADATA_TABLES to the list of ignored DDL in workload management. Fixes DCHECK failure when running Impala's full test suite with 'enable_workload_mgmt'. Change-Id: I69f7de9756aa730d70cd9187c9f869d5bcf67fce --- M be/src/service/workload-management.cc M tests/custom_cluster/test_query_log.py 2 files changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/21290/2 -- To view, visit http://gerrit.cloudera.org:8080/21290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69f7de9756aa730d70cd9187c9f869d5bcf67fce Gerrit-Change-Number: 21290 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12998: Add SHOW METADATA TABLES to ignored DDL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21290 ) Change subject: IMPALA-12998: Add SHOW_METADATA_TABLES to ignored DDL .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15900/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f7de9756aa730d70cd9187c9f869d5bcf67fce Gerrit-Change-Number: 21290 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 22:59:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21301 ) Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Apr 2024 21:39:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12963: Return parent PID when children spawned
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21278 ) Change subject: IMPALA-12963: Return parent PID when children spawned .. IMPALA-12963: Return parent PID when children spawned Returns the original PID for a command rather than any children that may be active. This happens during graceful shutdown in UBSAN tests. Also updates 'kill' to use the version of 'get_pid' that logs details to help with debugging. Moves try block in test_query_log.py to after client2 has been initialized. Removes 'drop table' on unique_database, since test suite already handles cleanup. Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Reviewed-on: http://gerrit.cloudera.org:8080/21278 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/common/impala_cluster.py M tests/custom_cluster/test_query_log.py 2 files changed, 72 insertions(+), 61 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Gerrit-Change-Number: 21278 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12963: Return parent PID when children spawned
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21278 ) Change subject: IMPALA-12963: Return parent PID when children spawned .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I214e79507c717340863d27f68f6ea54c169e4090 Gerrit-Change-Number: 21278 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 15 Apr 2024 22:00:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12998: Add SHOW METADATA TABLES to ignored DDL
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21290 ) Change subject: IMPALA-12998: Add SHOW_METADATA_TABLES to ignored DDL .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21290/1/be/src/service/workload-management.cc File be/src/service/workload-management.cc: http://gerrit.cloudera.org:8080/#/c/21290/1/be/src/service/workload-management.cc@231 PS1, Line 231: SHOW_METADATA_TABLES > Please add a corresponding test case to assert these queries are not includ Done -- To view, visit http://gerrit.cloudera.org:8080/21290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f7de9756aa730d70cd9187c9f869d5bcf67fce Gerrit-Change-Number: 21290 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 22:36:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12990: Fix impala-shell handling of unset rows deleted
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21284 ) Change subject: IMPALA-12990: Fix impala-shell handling of unset rows_deleted .. Patch Set 6: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/21284/6/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/21284/6/shell/impala_client.py@315 PS6, Line 315:in that order. If the DML operation doesn't return 'num_row_errors', then the third Comment should probably also address unset num_deleted_rows -> None as well. http://gerrit.cloudera.org:8080/#/c/21284/6/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/21284/6/shell/impala_shell.py@1366 PS6, Line 1366: return "Time elapsed: %2.2fs" % time_elapsed Are there tests covering this? Manual testing suggests it should work in Python 2 and 3 at least. -- To view, visit http://gerrit.cloudera.org:8080/21284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 Gerrit-Change-Number: 21284 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 15 Apr 2024 22:43:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13005: Create Query Live table in HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21302 ) Change subject: IMPALA-13005: Create Query Live table in HMS .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/15899/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 Gerrit-Change-Number: 21302 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 22:42:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13005: Create Query Live table in HMS
Hello Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21302 to look at the new patch set (#2). Change subject: IMPALA-13005: Create Query Live table in HMS .. IMPALA-13005: Create Query Live table in HMS Creates the 'sys.impala_query_live' table in HMS using a similar 'CREATE TABLE' command to 'sys.impala_query_log'. Updates frontend to identify a System Table based on the '__IMPALA_SYSTEM_TABLE' property. Tables improperly marked with '__IMPALA_SYSTEM_TABLE' will error when attempting to scan them because no relevant scanner will be available. Creating the table in HMS simplifies supporting 'SHOW CREATE TABLE' and 'DESCRIBE EXTENDED', so allows them for parity with Query Log. Explicitly disables 'COMPUTE STATS' on system tables as it doesn't work correctly. Updates workload management implementation to rely more on SystemTables.thrift definition, and adds DCHECKs to verify completeness and ordering. Testing: - adds additional test cases for changes to introspection commands - passes existing test_query_live and test_query_log suites Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/system-table-scanner.cc M be/src/service/workload-management-fields.cc M be/src/service/workload-management.cc M be/src/service/workload-management.h M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java A fe/src/main/java/org/apache/impala/analysis/SystemTableRef.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/SystemTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java D fe/src/test/java/org/apache/impala/catalog/SystemTableTest.java M tests/custom_cluster/test_query_live.py 16 files changed, 299 insertions(+), 342 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/21302/2 -- To view, visit http://gerrit.cloudera.org:8080/21302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 Gerrit-Change-Number: 21302 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13005: Create Query Live table in HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21302 ) Change subject: IMPALA-13005: Create Query Live table in HMS .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15901/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 Gerrit-Change-Number: 21302 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 23:20:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13005: Create Query Live table in HMS
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21302 Change subject: IMPALA-13005: Create Query Live table in HMS .. IMPALA-13005: Create Query Live table in HMS Creates the 'sys.impala_query_live' table in HMS using a similar 'CREATE TABLE' command to 'sys.impala_query_log'. Updates frontend to identify a System Table based on the '__IMPALA_SYSTEM_TABLE' property. Tables improperly marked with '__IMPALA_SYSTEM_TABLE' will error when attempting to scan them because no relevant scanner will be available. Creating the table in HMS simplifies supporting 'SHOW CREATE TABLE' and 'DESCRIBE EXTENDED', so allows them for parity with Query Log. Explicitly disables 'COMPUTE STATS' on system tables as it doesn't work correctly. Updates workload management implementation to rely more on SystemTables.thrift definition, and adds DCHECKs to verify completeness and ordering. Testing: - adds additional test cases for changes to introspection commands - passes existing test_query_live and test_query_log suites Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/system-table-scanner.cc M be/src/service/workload-management-fields.cc M be/src/service/workload-management.cc M be/src/service/workload-management.h M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java A fe/src/main/java/org/apache/impala/analysis/SystemTableRef.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/SystemTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java D fe/src/test/java/org/apache/impala/catalog/SystemTableTest.java M tests/custom_cluster/test_query_live.py 16 files changed, 299 insertions(+), 342 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/21302/1 -- To view, visit http://gerrit.cloudera.org:8080/21302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idf302ee54a819fdee2db0ae582a5eeddffe4a5b4 Gerrit-Change-Number: 21302 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12998: Add SHOW METADATA TABLES to ignored DDL
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21290 ) Change subject: IMPALA-12998: Add SHOW_METADATA_TABLES to ignored DDL .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21290/1/be/src/service/workload-management.cc File be/src/service/workload-management.cc: http://gerrit.cloudera.org:8080/#/c/21290/1/be/src/service/workload-management.cc@231 PS1, Line 231: SHOW_METADATA_TABLES Please add a corresponding test case to assert these queries are not included in the completed queries table. -- To view, visit http://gerrit.cloudera.org:8080/21290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f7de9756aa730d70cd9187c9f869d5bcf67fce Gerrit-Change-Number: 21290 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 18:38:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12737: List columns in profile and query history
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21142 ) Change subject: IMPALA-12737: List columns in profile and query history .. Patch Set 26: (5 comments) http://gerrit.cloudera.org:8080/#/c/21142/26//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21142/26//COMMIT_MSG@9 PS26, Line 9: "Select Columns", "Where Columns", "Join Columns", "Aggregate : Columns", and "OrderBy Columns" Should this add "Having Columns" as well? http://gerrit.cloudera.org:8080/#/c/21142/26/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/21142/26/common/thrift/Frontend.thrift@705 PS26, Line 705: // Columns referenced in a select list. : 21: optional list select_columns : : // Columns referenced in a where clause. : 22: optional list where_columns : : // Columns referenced in a join clause. : 23: optional list join_columns : : // Columns referenced in an aggregation. : 24: optional list aggregate_columns : : // Columns referenced in an order by clause. : 25: optional list orderby_columns I think there should be limit on maximum column names to log, or total length of the strings. If selecting hundreds of columns, it is probably not useful to log everything. http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@4488 PS26, Line 4488: Stream Is there benefit of passing Stream argument instead of Set? To me, Set is final, and it is caller responsibility to do all necessary transformation and filtering. http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@383 PS26, Line 383: Stream nonStarItems = : selectList_.getItems().stream().filter(elem -> !elem.isStar()); : nonStarItems.forEach(item -> item.getExpr().collect(SlotRef.class, slotRefs)); Can be combined into one? http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@387 PS26, Line 387: .filter(path -> path != null) Why is the filtering applied here and not in the concatenated Stream? -- To view, visit http://gerrit.cloudera.org:8080/21142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Gerrit-Change-Number: 21142 Gerrit-PatchSet: 26 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 19:32:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12737: List columns in profile and query history
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21142 ) Change subject: IMPALA-12737: List columns in profile and query history .. Patch Set 26: (5 comments) http://gerrit.cloudera.org:8080/#/c/21142/26//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21142/26//COMMIT_MSG@9 PS26, Line 9: "Select Columns", "Where Columns", "Join Columns", "Aggregate : Columns", and "OrderBy Columns" > Should this add "Having Columns" as well? I thought it was only used with "GROUP BY", but looks like it can be used separately. Currently Aggregate serves as an umbrella for both. I'll consider it. http://gerrit.cloudera.org:8080/#/c/21142/26/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/21142/26/common/thrift/Frontend.thrift@705 PS26, Line 705: // Columns referenced in a select list. : 21: optional list select_columns : : // Columns referenced in a where clause. : 22: optional list where_columns : : // Columns referenced in a join clause. : 23: optional list join_columns : : // Columns referenced in an aggregation. : 24: optional list aggregate_columns : : // Columns referenced in an order by clause. : 25: optional list orderby_columns > I think there should be limit on maximum column names to log, or total leng Perhaps we don't want that in the profile. I think in the query log table the intention is to be exhaustive. http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@4488 PS26, Line 4488: Stream > Is there benefit of passing Stream argument instead of Set? Mostly avoiding allocating another Set. But I do agree it's an unusual pattern. http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@383 PS26, Line 383: Stream nonStarItems = : selectList_.getItems().stream().filter(elem -> !elem.isStar()); : nonStarItems.forEach(item -> item.getExpr().collect(SlotRef.class, slotRefs)); > Can be combined into one? Will do. http://gerrit.cloudera.org:8080/#/c/21142/26/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@387 PS26, Line 387: .filter(path -> path != null) > Why is the filtering applied here and not in the concatenated Stream? Probably not needed for concatenated stream, but wouldn't hurt. There are a few test failures I need to look into that could be related. -- To view, visit http://gerrit.cloudera.org:8080/21142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Gerrit-Change-Number: 21142 Gerrit-PatchSet: 26 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 15 Apr 2024 23:05:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15903/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 02:06:13 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC
tables
..
WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset
of TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries
for external JDBC tables with Impala-Impala federation.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TODO
- run TPCDS queries in exhaustive mode.
- set proper default values for maxTotal and maxWaitMillis of DBCP
configuration parameters.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching
expected results for some decimal values.
- Coordinator open multiple JDBC connections in parallel for some
complex TPCDS quries with multiple DataSource scan nodes in
query plan, which cause connection not available from connection
pool if maxTotal is less than total number of scan nodes.
- maxWaitMillis is not working.
Testing:
- TODO Pass all TPCH/TCPDS queries.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
10 files changed, 1,532 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/2
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12874: Identify active and standby catalog and statestore the web debug endpoint
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/21294 )
Change subject: IMPALA-12874: Identify active and standby catalog and
statestore the web debug endpoint
..
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21294/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21294/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-12874: Identify active and standby catalog and statestore
the web debug endpoint
Incomplete sentence:
"Identify active and standby catalog and statestore daemon in the web debug
endpoint"
http://gerrit.cloudera.org:8080/#/c/21294/2//COMMIT_MSG@20
PS2, Line 20: Manually tests the web page, and verified the status display is
: correct.
If there is a failover does the status get updated? Can we also test that?
http://gerrit.cloudera.org:8080/#/c/21294/2/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/21294/2/be/src/util/default-path-handlers.cc@271
PS2, Line 271: } else {
admissiond also uses DaemonEnv, we should handle that case also. Maybe exit
early in case of admissiond?
--
To view, visit http://gerrit.cloudera.org:8080/21294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9435ba7a9549ea56f9d080a9315aecbcc630cd2
Gerrit-Change-Number: 21294
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Tue, 16 Apr 2024 02:37:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12152: Add query option to wait for events sync up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20131 ) Change subject: IMPALA-12152: Add query option to wait for events sync up .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15905/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a Gerrit-Change-Number: 20131 Gerrit-PatchSet: 19 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 03:07:26 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15902/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Apr 2024 01:37:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12152: Add query option to wait for events sync up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20131 ) Change subject: IMPALA-12152: Add query option to wait for events sync up .. Patch Set 19: (1 comment) http://gerrit.cloudera.org:8080/#/c/20131/19/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20131/19/be/src/catalog/catalog-server.cc@438 PS19, Line 438: VLOG_RPC << "WaitForHmsEvent(): request=" << ThriftDebugString(req); Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead. -- To view, visit http://gerrit.cloudera.org:8080/20131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a Gerrit-Change-Number: 20131 Gerrit-PatchSet: 19 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 02:43:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 14: Code-Review+2 looks good! -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 05:48:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#14). Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. IMPALA-12933: Avoid fetching unneccessary events of unwanted types There are several places where catalogd will fetch all events of a specific type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. To avoid bringing too much computation overhead to HMS's underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag, common_hms_event_types, is added to specify the common HMS event types. Once HIVE-28146 is resolved, we can set the wanted types directly in the HMS RPC and this approach can be simplified. UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most common unused events for Impala. They are also added to the default skip list. A new flag, default_skipped_hms_event_types, is added to configure this list. This patch also fixes an issue that events of the non-default catalog are not filtered out. In a local perf test, I generated 100K RELOAD events after creating a table in Hive. Then use the table in Impala to trigger metadata loading on it which will fetch the latest CREATE_TABLE event by polling all events after the last known CREATE_TABLE event. Before this patch, fetching the events takes 1s779ms. Now it takes only 395.377ms. Note that in prod env, the event messages are usually larger, we could have a larger speedup. Tests: - Added an FE test - Ran CORE tests Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 15 files changed, 313 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/14 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11495: Add glibc version and effective locale to the Web UI
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21252 ) Change subject: IMPALA-11495: Add glibc version and effective locale to the Web UI .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21252/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21252/2//COMMIT_MSG@9 PS2, Line 9: catalog, : coordinator and statestore nit: It sounds like executors don't have this. We can just remove this list. -- To view, visit http://gerrit.cloudera.org:8080/21252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia69c4d63df4beae29f5261691a8dcdd04b931de7 Gerrit-Change-Number: 21252 Gerrit-PatchSet: 2 Gerrit-Owner: Saurabh Katiyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Saurabh Katiyal Gerrit-Comment-Date: Tue, 16 Apr 2024 04:53:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12980: Translate CpuAsk into admission control slots
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21257 ) Change subject: IMPALA-12980: Translate CpuAsk into admission control slots .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/21257/12/tests/query_test/test_processing_cost.py File tests/query_test/test_processing_cost.py: http://gerrit.cloudera.org:8080/#/c/21257/12/tests/query_test/test_processing_cost.py@22 PS12, Line 22: from tests.common.test_dimensions import ( > flake8: F401 'tests.common.test_dimensions.add_mandatory_exec_option' impor Done -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 13 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Apr 2024 18:48:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 1: (6 comments) http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py File testdata/bin/create-tpc-jdbc-tables.py: http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py@41 PS1, Line 41: def drop_tables(): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py@47 PS1, Line 47: def create_tables(): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py@58 PS1, Line 58: flake8: E202 whitespace before '}' http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py@74 PS1, Line 74: def get_test_file_path(workload): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/21304/1/testdata/bin/create-tpc-jdbc-tables.py@83 PS1, Line 83: if __name__ == "__main__": flake8: E305 expected 2 blank lines after class or function definition, found 1 http://gerrit.cloudera.org:8080/#/c/21304/1/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/1/tests/query_test/test_tpcds_queries.py@1119 PS1, Line 1119: d flake8: E301 expected 1 blank line, found 0 -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Apr 2024 01:14:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 14: (3 comments) Refactored some lengthy codes. http://gerrit.cloudera.org:8080/#/c/21186/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21186/13//COMMIT_MSG@27 PS13, Line 27: and t > nit: and this approach can be simplified Done http://gerrit.cloudera.org:8080/#/c/21186/13//COMMIT_MSG@31 PS13, Line 31: A new flag, default_skipped_hms_event_types, is added to configure : this > This seems like a duplicate after line 24 Oops, wrote a wrong name here.. http://gerrit.cloudera.org:8080/#/c/21186/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21186/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@361 PS13, Line 361: : // Total number of dbs, tables and functions in the catalog cache. : // Updated in each catalog topic update (getCatalogDelta()). : private int numDbs_ = 0; : private int numTables_ = 0; > This comment could get stale easily - maybe point to the flag instead? Done -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 01:42:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12933: Avoid fetching unneccessary events of unwanted types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted types .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10543/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 16 Apr 2024 01:42:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12152: Add query option to wait for events sync up
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Michael Smith, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20131 to look at the new patch set (#19). Change subject: IMPALA-12152: Add query option to wait for events sync up .. IMPALA-12152: Add query option to wait for events sync up Event-processor is designed to get rid of manual RT/IM (RefreshTable / InvalidateMetadata) commands that sync up with external HMS modifications. However, event processing could be delayed. Queries might still see stale metadata if the event-processor is lagging behind. This patch adds a mechanism to let query planning wait until the metadata is synced up. To be specific, coordinator will not start planning until the last synced event id of catalogd reaches the latest event id when the query is submitted. A new catalogd RPC, WaitForHmsEvent, is added for this. Coordinator parses the query and sends potential dbs/tables that are required by the query. Catalogd records the latest event id and returns the required catalog updates once it catches up with that event id. Coordinator then applies the catalog updates and continues query planning. Note that the current implementation waits for the latest event id when the WaitForHmsEvent RPC is received at catalogd side. We can improve it once HIVE-27499 is resolved, so we can efficiently detect whether some given dbs/tables have unsynced events and just wait for the *largest* id of them. Dbs/tables without unsynced events don't need to block query planning. A new query option, sync_hms_events_wait_time_s, is added to configure the timeout for waiting. It's 0 by default, which disables the waiting mechanism. Users can turn it on for sensitive queries that depend on external modifications. Another new query option, sync_hms_events_strict_mode, is added to control the behavior on errors, e.g. timeout or event-processor in error state. It defaults to false (non-strict mode). In the strict mode, coordinator will fail the query if it fails to wait for HMS events to be synced in catalogd. In the non-strict mode, coordinator will start planning with a warning message in profile (and in client outputs if the client consumes the get_log results, e.g. in impala-shell). Example usage - query the table after inserting into dynamic partitions in Hive. We don't know what partitions are modified so running REFRESH in Impala is inefficient since it reloads all partitions. hive> insert into tbl partition(p) select * from tbl2; impala> set sync_hms_events_wait_time_s=300; impala> select * from tbl; With this new feature, let catalogd reload the updated partitions based on HMS events, which is more efficient than REFRESH. The wait time can be set to the largest lag of event processing that has been observed in the cluster. Some timeline items are added in query profile for this waiting, e.g. A succeeded wait: Query Compilation: 937.279ms - Synced events from Metastore: 909.162ms (909.162ms) - Metadata of all 1 tables cached: 911.005ms (1.843ms) - Analysis finished: 919.600ms (8.595ms) A failed wait: Query Compilation: 1s321ms - Failed to sync events from Metastore: 40.883ms (40.883ms) - Metadata load started: 41.618ms (735.633us) For better debuggability in tests, add logs in run_stmt_in_hive to print the Hive statements. Limitation: Strict mode might fail in timeout if the latest event is skipped by the event processor, thus the last synced event id won't reach the latest event it until there are new events generated in HMS. Currently, OPEN_TXN events are skipped by event processor. Tests - Add test to verify planning waits until catalogd is synced with HMS changes. - Add test on the error handling when HMS event processing is disabled - There are some existing tests that use EventProcessorUtils.wait_for_event_processing() to wait until events synced. Modify them to use the new query option in queries need this. Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/runtime/coordinator.cc M be/src/service/fe-support.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDropRoleStmt.java M
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21304
Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC
tables
..
WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset
of TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries
for external JDBC tables with Impala-Impala federation.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TODO
- run TPCDS queries in exhaustive mode.
- set proper default values for maxTotal and maxWaitMillis of DBCP
configuration parameters.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching
expected results for some decimal values.
- Coordinator open multiple JDBC connections in parallel for some
complex TPCDS quries with multiple DataSource scan nodes in
query plan, which cause connection not available from connection
pool if maxTotal is less than total number of scan nodes.
- maxWaitMillis is not working.
Testing:
- TODO Pass all TPCH/TCPDS queries.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
10 files changed, 1,527 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/1
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: WIP IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15904/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Apr 2024 02:59:34 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-3.4.2) IMPALA-12999: Add log4j.properties to the DEB/RPM packages
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21299 ) Change subject: IMPALA-12999: Add log4j.properties to the DEB/RPM packages .. IMPALA-12999: Add log4j.properties to the DEB/RPM packages log4j.properties is required to configure log4j before logs from it are redirected to glog (done in GlogAppender#Install()). This is crucial to show error logs during initialization, especially while lauching the JVM. See the JIRA description for an example. This copies log4j.properties from fe/src/test/resources directly since it hasn't changed for years. Change-Id: Iee0b9699ef313aa8e94bd351fa51fad3ea0cdf57 Reviewed-on: http://gerrit.cloudera.org:8080/21293 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/21299 Reviewed-by: Zihao Ye Tested-by: Quanlong Huang --- A package/conf/log4j.properties 1 file changed, 28 insertions(+), 0 deletions(-) Approvals: Zihao Ye: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/21299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-3.4.2 Gerrit-MessageType: merged Gerrit-Change-Id: Iee0b9699ef313aa8e94bd351fa51fad3ea0cdf57 Gerrit-Change-Number: 21299 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR](branch-3.4.2) IMPALA-12999: Add log4j.properties to the DEB/RPM packages
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21299 ) Change subject: IMPALA-12999: Add log4j.properties to the DEB/RPM packages .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-3.4.2 Gerrit-MessageType: comment Gerrit-Change-Id: Iee0b9699ef313aa8e94bd351fa51fad3ea0cdf57 Gerrit-Change-Number: 21299 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Mon, 15 Apr 2024 10:01:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12990: Fix impala-shell handling of unset rows deleted
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/21284 ) Change subject: IMPALA-12990: Fix impala-shell handling of unset rows_deleted .. Patch Set 5: Code-Review+2 Looks good, thank you Csaba! -- To view, visit http://gerrit.cloudera.org:8080/21284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5812b8006b9cacf34a7a0dbbc89a486d8b454438 Gerrit-Change-Number: 21284 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 15 Apr 2024 09:23:22 + Gerrit-HasComments: No
